Call for Semantic Web Challenges Proposals

Overview

The ESWC organizers are glad to announce that the Challenges Track will be included again in the program of ESWC 2017!  Eight challenges were held last year [1] and allowed the conference to attract a broader audience beyond the Semantic Web community, also spanning across disciplines such as Recommender Systems or Knowledge Extraction. 
This year, a call for challenges is open in order to allow the selection of challenges to be held at the conference.

The purpose of challenges is to showcase the maturity of state of the art methods and tools on tasks common to the Semantic Web community and adjacent disciplines, in a controlled setting involving rigorous evaluation.

Semantic Web Challenges are an official track of the conference, ensuring significant visibility for the challenges as well as participants. Challenge participants are asked to present their submissions as well as provide a paper describing their work.  These papers must undergo a peer-review by experts relevant to the challenge task, and will be published in the challenge proceedings.

Next to the publication of proceedings, challenges at ESWC2017 benefit from high visibility and direct access to the ESWC audience and community.

Challenge Proposals

Challenge organizers are encouraged to submit proposals adhering to the following criteria:

• At least one task involving semantics in data. 
  The task(s) should be well defined and related to the Semantic Web but not necessarily confined to it. It is highly encouraged to consider tasks which involve other, highly related communities, such as NLP, Recommender Systems, Machine Learning or Information Retrieval. If multiple tasks are provided the tasks should be independent so that participants may choose which to participate in.
• Task descriptions are likely to be interesting to a wider audience. 
  We encourage the challenge organizers to propose at least one basic task that can be addressed by a larger audience from their community. Engaging with your challenge audience and obtaining feedback from your target group on the task design might be helpful for shaping the task and ensuring sufficient number of participants.
• Clear and rigorous definition of the tasks. 
  For each task, you should define a deterministic and objective way to verify if the goal of the task has been achieved, and to which extent it has been achieved (if applicable). The best way is usually to provide detailed examples of input data and expected output. The examples must cover all the possible situations that can occur while performing the task, and should leave no place to ambiguity about whether in a particular case the task is done or not.
• Valid dataset (if applicable). 
  If accepted, you should find or create a dataset that will be used for the challenge. In any case, you must specify the provenance of the dataset (if it contains human annotation – how were those obtained). You must make sure you have the right to use/publish this dataset and clearly state the license for its use within the challenge. The dataset should be split in two parts – the training part, and the evaluation part. The training part contains the data, and the results that should be obtained when performing the task. As for the evaluation part, you should only publish the data, and make sure that the correct results have not previously been available to the participants. When proposing the challenge you must provide details on the dataset and on the way it is/will be created – the dataset can be made available later.
• Challenge Committee. 
  Composed of at least 3 respected researchers with experience in the tasks of the challenge. They help evaluate the papers submitted by the participants, and also validate the evaluation procedure.
• Evaluation metrics and procedure. 
  For each task there must be a number of objective criteria (metrics), e.g. precision and recall. The evaluation procedure and the way in which the metrics will be calculated must be clearly specified and made transparent to participants.

Among the selection criteria for choosing the supported challenges are:

• Potential number to interested participants
• Rigor and transparency of the evaluation procedure
• Relevance for the Semantic Web community
• Endorsements (from researchers working on the task, from industry players interested in results, from future participants)

Important Dates

• Challenges proposals due: Friday November 18th, 2016 - 23:59 Hawaii Time
• Challenges chosen/merged – notification to organizers sent: Friday December 2nd, 2016
• Training data ready and challenges Calls for Papers: Friday January 13th, 2017
• Challenge papers submission deadline (5 pages document): Friday March 10th, 2017
• Challenge paper reviews: Tuesday April 5th, 2017
• Notifications sent to participants and invitations to submit task results: Friday April 7th, 2017
• Test data (and other participation tools) published: Friday April 7th, 2017
• Camera ready papers for the conference (5 pages document): Sunday April 23rd, 2017
• Submission of challenge results: free choice of organizers
• Proclamation of winners: During ESWC2017 closing ceremony
• Camera ready paper for the challenge post proceedings (15 pages document): Friday June 30th, 2017 (tentative deadline)

Submission Details

The challenges proposals should contain at least the following elements:

• A summary description of the challenge and tasks
• How the training/testing data will be built and/or procured
• The evaluation methodology to be used, including clear evaluation criteria and the exact way in which they will be measured. Who will perform the evaluation and how will transparency be assured?
• The anticipated availability of the necessary resources to the participants
• The resources required to prepare the tasks (computation and annotation time, costs of annotations, etc)
• The list of challenge committee members who will evaluate the challenge papers (please indicate which of the listed members already accepted the role)

In case of doubt, feel free to send us your challenge proposal drafts as early as possible – the challenges chairs will provide you with feedback and answers to questions you may have.

Please submit proposals via Easychair as soon as possible and no later than *18 November 2016*.

Challenges

7th Open Challenge on Question Answering over Linked Data (QALD-7)

The past years have seen a growing amount of research on question answering (QA) over Semantic Web data, shaping an interaction paradigm that allows end users to profit from the expressive power of Semantic Web standards while at the same time hiding their complexity behind an intuitive and easy-to-use interface. 
At the same time the growing amount of data has led to a heterogeneous data landscape where QA systems struggle to keep up with the volume, variety and veracity of the underlying knowledge.
The Question Answering over Linked Data (QALD) challenge aims at providing an up-to-date benchmark for assessing and comparing state-of-the-art-systems that mediate between a user, expressing his or her information need in natural language, and RDF data. It thus targets all researchers and practitioners working on natural language processing for question answering, multilingual information retrieval etc. The main goal is to gain insights into the different approaches and into possible solutions for coping with the large, heterogeneous and distributed nature of Semantic Web data.
Since many of the topics relevant for QA over Linked Data lie at the core of ESWC (Multilinguality, Semantic Web, Human-Machine-Interfaces), we will  run the 7th instantiation of QALD again at ESWC 2017. 

https://project-hobbit.eu/challenges/qald2017/

ESWC-17 Challenge on Semantic Sentiment Analysis

Social media evolution has given users one important opportunity for expressing their thoughts and opinions online. The information thus produced is related to many different areas such as commerce, tourism, education, health and causes the size of the Social Web to expand exponentially.
Mining opinions and sentiments from natural language involves a deep understanding of most of the explicit and implicit, regular and irregular, syntactical and semantic rules proper of a language.
Existing approaches are mainly focused on the identification of parts of the text where opinions and sentiments can be explicitly expressed such as polarity terms, expressions, affect words. They usually adopt purely syntactical approaches and are heavily dependent on the source language of the input text. They miss many language patterns where opinions can be expressed though.
Understanding the semantics of a sentence offers an exciting research opportunity and challenge to the Semantic Web community as well. In fact, the Semantic Sentiment Analysis Challenge aims to go beyond a mere word-level analysis of text and provides novel methods to opinion mining and analysis that can transform more efficiently unstructured textual information to structured machine-processable data.
By relying on Semantic Web best practices and techniques, fine-grained sentiment analysis relies on the implicit features associated with natural language concepts. Unlike purely syntactical techniques, semantic sentiment analysis approaches are able to detect also sentiments that are implicitly expressed within the text.

https://github.com/diegoref/SSAC2017

The Mighty Storage Challenge

The aim of this challenge is to test the performance of solutions for SPARQL processing in aspects that are relevant for modern applications. These include ingesting data, answering queries on large datasets and serving as backend for applications driven by Linked Data. The challenge will test the systems against data derived from real applications and with realistic loads. This year, the challenge will  comprise the following tasks: 

• Task 1 (Data Ingestion) will measure how well systems can ingest streams of RDF data. 
• Task 2 (Data Storage) will measure how data stores perform with different types of queries. 
• Task 3 (Versioning) will measure how well versioning and archiving systems for Linked Data perform when they store multiple versions of large data sets. 
• Task 4 (Browsing) will check existing solutions for how well they support applications that need browsing through large data sets. 

A full description of the challenge as well as the training data can be found at the challenge web page at:

https://project-hobbit.eu/challenges/mighty-storage-challenge/

Open Knowledge Extraction Challenge 2017

The Open Knowledge Extraction Challenge invites researchers and practitioners from academia as well as industry to compete to the aim of pushing further the state of the art of knowledge extraction from text for the Semantic Web. The challenge has the ambition to provide a reference framework for research in this field by redefining a number of tasks typically from information and knowledge extraction by taking into account Semantic Web requirements and has the goal to test the performance of knowledge extraction systems. This year, the challenge goes in the third round and consists of four tasks which include named entity identification, typing and disambiguation by linking to a knowledge base as well as knowledge extraction of RDF triples with respect to the DBpedia knowledge base. A competing system has to process two datasets for a task.  One small dataset that consists of manually curated documents and another large dataset that consists of synthetic documents. The performance measure of a participating system is twofold base on (1) precision and recall and on (2) precision and recall with respect to the runtime of the system.

https://project-hobbit.eu/challenges/oke2017-challenge-eswc-2017

​Semantic Publishing Challenge 2017

The 4th edition of the Semantic Publishing Challenge continues pursuing the objective of assessing the quality of scientific output in its ecosystem, evolving the dataset bootstrapped in 2014 and continued in 2015 and 2016. This year’s edition has a similar structure as the previous three editions with certain modifications, which reflect the lessons we have learned from the previous editions in our in-depth reviews.
This edition looks into (i) extracting information from the PDF sources of full papers in workshop proceedings and enriching this information with knowledge from existing datasets; (ii) interlinking the aforementioned output, as well as the CEUR-WS.org open access workshop proceedings dataset, produced by the winning solutions of previous challenge editions, to other datasets with related context. Thus, a combination of broadly investigated technologies, such as Information and PDF Extraction, Natural Language Processing and, in particular, Named Entity Recognition, as well as Instance Matching and Link Discovery, is required to deal with the challenge’s tasks.
Participants will submit (i) their dataset along with (ii) standard-conforming SPARQL queries against the dataset, and (iii) CSV-formatted query results, as well as (iv) metadata for the submitted datasets.
We will measure the extent to which the submitted dataset refines the CEUR-WS.org dataset by determining the precision and recall of predefined evaluation queries. Last but not least, generating metadata will also be promoted and evaluated by our evaluation. Participants are invited to submit new solutions or to refine and enrich their solutions produced in the past.

https://github.com/ceurws/lod/wiki/SemPub2017

For any questions, please do not hesitate to get in touch with the ESWC2017 challenge chairs

• Mauro Dragoni, Fondazione Bruno Kessler, Italy (dragoni@fbk.eu)
• Monica Solanki, University of Oxford, United Kingdom (monika.solanki@cs.ox.ac.uk)

[1] http://2016.eswc-conferences.org/call-challenges