James Fan

In this paper, we present a novel approach for relation extraction using only term
pairs as the input without textual features. We aim to build a single joint space for each
relation which is then used to produce relation specific term embeddings. The proposed method fits particularly well for domains in which similar arguments are often associated with similar relations. It can also handle the situation when the labeled data is limited. The proposed method is evaluated both theoretically… Show more

In this paper, we present a novel approach for relation extraction using only term
pairs as the input without textual features. We aim to build a single joint space for each
relation which is then used to produce relation specific term embeddings. The proposed method fits particularly well for domains in which similar arguments are often associated with similar relations. It can also handle the situation when the labeled data is limited. The proposed method is evaluated both theoretically with a proof for the closed-form solution and experimentally with promising results on both DBpedia and medical relations. Show less

Poker is a family of card games that includes many variations. We hypothesize that most poker games can be solved as a pattern matching problem, and propose creating a strong poker playing system based on a unified poker representation. Our poker player learns through iterative self-play, and improves its understanding of the game by training on the results of its previous actions without sophisticated domain knowledge. We evaluate our system on three poker games:… Show more

Poker is a family of card games that includes many variations. We hypothesize that most poker games can be solved as a pattern matching problem, and propose creating a strong poker playing system based on a unified poker representation. Our poker player learns through iterative self-play, and improves its understanding of the game by training on the results of its previous actions without sophisticated domain knowledge. We evaluate our system on three poker games: single player video poker, two-player Limit Texas Hold’em, and finally two-player 2-7 triple draw poker. We show that our model can quickly learn patterns in these very different poker games while it improves from zero knowledge to a competitive player against human experts. The contributions of this paper include: (1) a novel representation for poker games, extendable to different poker variations, (2) a Convolutional Neural Network (CNN) based learning model that can effectively learn the patterns in three different games, and (3) a self-trained system that significantly beats the heuristic-based program on which it is trained, and our system is competitive against human expert players. Show less

In this paper, we present a manifold model for medical relation extraction. Our model
is built upon a medical corpus containing 80M sentences (11 gigabyte text) and designed
to accurately and efficiently detect the key medical relations that can facilitate
clinical decision making. Our approach integrates domain specific parsing and typing
systems, and can utilize labeled as well as unlabeled examples. To provide users
with more flexibility, we also take label weight into… Show more

In this paper, we present a manifold model for medical relation extraction. Our model
is built upon a medical corpus containing 80M sentences (11 gigabyte text) and designed
to accurately and efficiently detect the key medical relations that can facilitate
clinical decision making. Our approach integrates domain specific parsing and typing
systems, and can utilize labeled as well as unlabeled examples. To provide users
with more flexibility, we also take label weight into consideration. Effectiveness
of our model is demonstrated both theoretically with a proof to show that the solution
is a closed-form solution and experimentally with positive results in experiments.
Show less

Recent years have seen significant advances on the creation of large-scale knowledge bases (KBs). Extracting knowledge from Web pages, and integrating it into a coherent KB is a task that spans the areas of natural language processing, information extraction, information integration, databases, search and machine learning. Some of the latest developments in the field were presented at the AKBC-WEKEX workshop on knowledge extraction at the NAACL-HLC 2012 conference. This workshop included 23… Show more

Recent years have seen significant advances on the creation of large-scale knowledge bases (KBs). Extracting knowledge from Web pages, and integrating it into a coherent KB is a task that spans the areas of natural language processing, information extraction, information integration, databases, search and machine learning. Some of the latest developments in the field were presented at the AKBC-WEKEX workshop on knowledge extraction at the NAACL-HLC 2012 conference. This workshop included 23 accepted papers, and 11 keynotes by senior researchers. The workshop had speakers from all major search engine providers, government institutions, and the leading universities in the field. In this survey, we summarize the papers, the keynotes, and the
discussions at this workshop. Show less

Major advances in Question Answering technology were needed for IBM Watson1
to play Jeopardy!2 at championship level – the show requires rapid-fire answers to challenging
natural language questions, broad general knowledge, high precision, and accurate confi-
dence estimates. In addition, Jeopardy! features four types of decision making carrying
great strategic importance: (1) Daily Double wagering; (2) Final Jeopardy wagering; (3)
selecting the next square when in control of the… Show more

Major advances in Question Answering technology were needed for IBM Watson1
to play Jeopardy!2 at championship level – the show requires rapid-fire answers to challenging
natural language questions, broad general knowledge, high precision, and accurate confi-
dence estimates. In addition, Jeopardy! features four types of decision making carrying
great strategic importance: (1) Daily Double wagering; (2) Final Jeopardy wagering; (3)
selecting the next square when in control of the board; (4) deciding whether to attempt
to answer, i.e., “buzz in.” Using sophisticated strategies for these decisions, that properly
account for the game state and future event probabilities, can significantly boost a player’s
overall chances to win, when compared with simple “rule of thumb” strategies.
This article presents our approach to developing Watson’s game-playing strategies,
comprising development of a faithful simulation model, and then using learning and MonteCarlo
methods within the simulator to optimize Watson’s strategic decision-making. After
giving a detailed description of each of our game-strategy algorithms, we then focus in
particular on validating the accuracy of the simulator’s predictions, and documenting performance
improvements using our methods. Quantitative performance benefits are shown
with respect to both simple heuristic strategies, and actual human contestant performance
in historical episodes. We further extend our analysis of human play to derive a number of
valuable and counterintuitive examples illustrating how human contestants may improve
their performance on the show. Show less

Questions often explicitly request a particular type of answer. One
popular approach to answering natural language questions involves filtering candidate
answers based on precompiled lists of instances of common answer types
(e.g., countries, animals, foods, etc.). Such a strategy is poorly suited to an open
domain in which there is an extremely broad range of types of answers, and the
most frequently occurring types cover only a small fraction of all answers. In
this paper we… Show more

Questions often explicitly request a particular type of answer. One
popular approach to answering natural language questions involves filtering candidate
answers based on precompiled lists of instances of common answer types
(e.g., countries, animals, foods, etc.). Such a strategy is poorly suited to an open
domain in which there is an extremely broad range of types of answers, and the
most frequently occurring types cover only a small fraction of all answers. In
this paper we present an alternative approach called TyCor, that employs soft filtering
of candidates using multiple strategies and sources. We find that TyCor
significantly outperforms a single-source, single-strategy hard filtering approach,
demonstrating both that multi-source multi-strategy outperforms a single source,
single strategy, and that its fault tolerance yields significantly better performance
than a hard filter. Show less

Access to a large amount of knowledge is critical for success at
answering open-domain questions for DeepQA systems such as
IBM Watsoni. Formal representation of knowledge has the
advantage of being easy to reason with, but acquisition of structured
knowledge in open domains from unstructured data is often difficult
and expensive. Our central hypothesis is that shallow syntactic
knowledge and its implied semantics can be easily acquired and can
be used in many areas of a… Show more

Access to a large amount of knowledge is critical for success at
answering open-domain questions for DeepQA systems such as
IBM Watsoni. Formal representation of knowledge has the
advantage of being easy to reason with, but acquisition of structured
knowledge in open domains from unstructured data is often difficult
and expensive. Our central hypothesis is that shallow syntactic
knowledge and its implied semantics can be easily acquired and can
be used in many areas of a question-answering system. We take a
two-stage approach to extract the syntactic knowledge and implied
semantics. First, shallow knowledge from large collections of
documents is automatically extracted. Second, additional
semantics are inferred from aggregate statistics of the
automatically extracted shallow knowledge. In this paper, we
describe in detail what kind of shallow knowledge is extracted,
how it is automatically done from a large corpus, and how
additional semantics are inferred from aggregate statistics. We
also briefly discuss the various ways extracted knowledge is used
throughout the IBM DeepQA system. Show less

A key phase in the DeepQA architecture is Hypothesis Generation, in
which candidate system responses are generated for downstream
scoring and ranking. In the IBM Watsoni system, these hypotheses
are potential answers to Jeopardy!i questions and are generated by
two components: search and candidate generation. The search
component retrieves content relevant to a given question from
Watson’s knowledge resources. The candidate generation component
identifies potential answers… Show more

A key phase in the DeepQA architecture is Hypothesis Generation, in
which candidate system responses are generated for downstream
scoring and ranking. In the IBM Watsoni system, these hypotheses
are potential answers to Jeopardy!i questions and are generated by
two components: search and candidate generation. The search
component retrieves content relevant to a given question from
Watson’s knowledge resources. The candidate generation component
identifies potential answers to the question from the retrieved
content. In this paper, we present strategies developed to use
characteristics of Watson’s different knowledge sources and to
formulate effective search queries against those sources. We further
discuss a suite of candidate generation strategies that use various
kinds of metadata, such as document titles or anchor texts in
hyperlinked documents. We demonstrate that a combination of these
strategies brings the correct answer into the candidate answer
pool for 87.17% of all the questions in a blind test set, facilitating
high end-to-end question-answering performance. Show less

This paper aims to solve the problem of improving the ranking of
answer candidates for factoid based questions in a state-of-the-art
Question Answering system. We first provide an extensive comparison
of 5 ranking algorithms on two datasets – from the Jeopardy
quiz show and a medical domain. We then show the effectiveness
of a cascading approach, where the ranking produced by one
ranker is used as input to the next stage. The cascading approach
shows sizeable gains on both… Show more

This paper aims to solve the problem of improving the ranking of
answer candidates for factoid based questions in a state-of-the-art
Question Answering system. We first provide an extensive comparison
of 5 ranking algorithms on two datasets – from the Jeopardy
quiz show and a medical domain. We then show the effectiveness
of a cascading approach, where the ranking produced by one
ranker is used as input to the next stage. The cascading approach
shows sizeable gains on both datasets. We finally evaluate several
rank aggregation techniques to combine these algorithms, and find
that Supervised Kemeny aggregation is a robust technique that always
beats the baseline ranking approach used by Watson for the
Jeopardy competition. We further corroborate our results on TREC
Question Answering datasets.
Show less

A key requirement for high-performing question-answering (QA)
systems is access to high-quality reference corpora from which
answers to questions can be hypothesized and evaluated. However,
the topic of source acquisition and engineering has received very
little attention so far. This is because most existing systems were
developed under organized evaluation efforts that included reference
corpora as part of the task specification. The task of answering
Jeopardy!i… Show more

A key requirement for high-performing question-answering (QA)
systems is access to high-quality reference corpora from which
answers to questions can be hypothesized and evaluated. However,
the topic of source acquisition and engineering has received very
little attention so far. This is because most existing systems were
developed under organized evaluation efforts that included reference
corpora as part of the task specification. The task of answering
Jeopardy!i questions, on the other hand, does not come with such a
well-circumscribed set of relevant resources. Therefore, it became
part of the IBM Watsoni effort to develop a set of well-defined
procedures to acquire high-quality resources that can effectively
support a high-performing QA system. To this end, we developed
three procedures, i.e., source acquisition, source transformation, and
source expansion. Source acquisition is an iterative development
process of acquiring new collections to cover salient topics deemed
to be gaps in existing resources based on principled error analysis.
Source transformation refers to the process in which information
is extracted from existing sources, either as a whole or in part, and is
represented in a form that the system can most easily use. Finally,
source expansion attempts to increase the coverage in the content of
each known topic by adding new information as well as lexical
and syntactic variations of existing information extracted from
external large collections. In this paper, we discuss the methodology
that we developed for IBM Watson for performing acquisition,
transformation, and expansion of textual resources. We demonstrate
the effectiveness of each technique through its impact on candidate
recall and on end-to-end QA performance. Show less