Research Projects NTT-MIT Research Collaboration: a partnership in the future of communication and computation

Multilingual Conversational System Research

9807-11

Start date: 07/98

James Glass and Stephanie Seneff
MIT LCS

Kiyoaki Aikawa
NTT

Project summary


A Japanese conversational system is being built that allows users to interact with weather services over the telephone.

Project description


 

The Spoken Language Systems group has been developing computer conversational systems for information access for over ten years. The Jupiter system for weather information is our most mature system, and has been available to the public via a toll-free telephone number for the past two years. The MIT/NTT collaboration project concerns the development of a version of the Jupiter system that communicates with the user in Japanese. It utilizes the same infrastructure as the English-based system, but recognizer vocabularies and grammar rules are provided for Japanese. The weather reports are obtained from multiple sites on the Web, and are represented in a generic "semantic frame" format. Generation rules map these meaning representations into Japanese. Ultimately, we expect to incorporate weather information available on the Web in Japanese for Japanese cities, which will augment our information sources for both the English-based and Japanese-based systems. We expect this research to help us understand how to build systems that can be ported to multiple languages efficiently and effectively.


Demos, movies and other examples



You can try out the English language version of Jupiter over the phone.



Speaking Style: Jupiter is intended to recognize and understand natural, conversational speech. When you talk to Jupiter, it is actually best if you speak naturally to the system, as you would to another person. In other words, you don't need to pause between words, overemphasize words (e.g., pronouncing them one syllable at a time), or speak in computerese (e.g. "weather boston" vs. "what's the weather in boston"). The system also doesn't do as well if you shout, mumble, or speak softly. It is best if you speak clearly, as you might to a young child. Also, if you speak softly (especially at the end of your sentence), or pause extensively, the end point detector might cause your speech to be clipped, which will make it harder for Jupiter to understand you.

Here are some example sentences you can speak to Jupiter. Notice that the system will remember some aspects of your previous queries.

- What cities do you know about in California?
- How about in Japan?
- What will the temperature be in Boston tomorrow?
- What about the humidity?
- Are there any flood warnings in the United States?
- Where is it sunny in the Caribbean?
- What's the wind speed in Chicago?
- How about London?
- Can you give me the forecast for Seattle?
- Will it rain tomorrow in Denver?

Here is how to call Jupiter.

  • When you call, you will be connected with Jupiter (the connection will take about 5 sec.), and the system will greet you with something like "Welcome to Jupiter - the automated weather service from MIT. How may I help you?"

  • At this point, you will hear a brief high pitch beep, indicating that the computer is listening. You can now ask your question, e.g., what's the forecast in Boston, will it rain tomorrow, when will the sun rise, etc.

  • The toll-free number for calling Jupiter from within North America is 1-888-573-TALK (1-888-573-8255).

  • If you wish to call Jupiter from outside North America, the number is +1-617-258-0300. This is not a toll free number, and you will be responsible for the (international) long-distance charges.

    The principal investigators


    Presentations and posters


    "Mokusei: Multilingual Conversational Interfaces", S. Seneff, J. Glass, T.J. Hazen, J. Polifroni, V. Zue and Y. Minami, NTT, Musashino, Japan, January 2000.

    "Multilingual Conversational Interfaces: An NTT Collaboration", Stephanie Seneff, NTT, Musashino, Japan, January 2000.

    Publications


    V. Zue, S. Seneff, J. Glass, J. Polifroni, C. Pao, T. Hazen, and L. Hetherington, "Jupiter: A telephone-based conversational interface for weather information," to appear in IEEE Trans. Speech and Audio Processing, Vol. 8, No. 1, January, 2000.

    S. Seneff, E. Hurley, R. Lau, C. Pao, P. Schmid, and V. Zue, "Galaxy-II: A reference architecture for conversational system development," in Proc. ICSLP 1998, 931-934, Sydney, Australia.

    V. Zue, S. Seneff, J. Polifroni, H. Meng and J. Glass, "Multilingual human-computer interactions: From information access to language learning," in Proc. ICSLP 1996, 2207-2210, Philadelphia, PA.

    J. Glass, G. Flammia, D. Goodine, M. Phillips, J. Polifroni, S. Sakai, S. Seneff, and V. Zue, "Multilingual Spoken-language Understanding in the MIT Voyager System," Speech Communication, vol. 17, 1-18, 1995.

    Proposals and progress reports


    Proposals:

    NTT Bi-Annual Progress Report, July to December 1998:

    NTT Bi-Annual Progress Report, January to June 1999:

    NTT Bi-Annual Progress Report, July to December 1999:

    NTT Bi-Annual Progress Report, January to June 2000:

    NTT Bi-Annual Progress Report, July to December 2000:

    NTT Bi-Annual Progress Report, January to June 2001:

    For more information