Xtext’s New Generator

Xtext 2.9 adds support for two additional editor platforms: Web-editors and IDEA. It also adds generic build system integration for Maven and for Gradle. As a result the number of generator options has grown a lot. So we took the opportunity to re-implement Xtext’s code generator – the one that creates the language infrastructure from the grammar.

The new generator no longer relies on Xpand, uses dependency injection to wire up the components and is a lot easier to configure for the user. It is still run by means of an MWE2 workflow. Don’t worry: The old generator is still there so you don’t have to migrate immediately. But new languages will automatically make use of the new generator.

The New Xtext Project Wizard

To understand the new code generator let us first have a look at the new New Xtext Project Wizard.

Xtext 2.9 new project Wizard

The first three facets stand for the three supported editor front-ends. The new Generic IDE Support is needed by all of them. It encapsulates platform-neutral editor functionality such as syntax coloring and parser based code completion. Note that you can skip the generation of an editor entirely by not checking any of the first four facets.

In addition to Testing Support, you can choose a preferred build system. If either Maven or Gradle is checked, the wizard will generate all the projects within a parent project to comply to the standard hierarchical file layout of a Maven/Gradle application. Selecting Maven/Gradle as Source Layout will create the common Maven file structure with source folders as src/main/java, src/test/java etc.

Note that not all combinations make sense, and you will be warned or even corrected by the wizard if your config is irregular.

The Project Layout

The resulting project layout for the above configuration will look like this:

Xtext 2.9 project layout

  • mydsl: The base runtime infrastructure of the language, like the parser, validation etc. Always generated.
  • mydsl.ide: The common editor functionality for all front-ends. Generated when Generic IDE Support is checked.
  • mydsl.idea: The Intellij IDEA plug-in.
  • mydsl.parent: The parent project containing all others. Generated if a Preferred Build System is chosen.
  • mydsl.target: Contains the target platform definition, i.e. the definition of the Eclipse plug-ins and features the code is compiled against. Needed when using Maven or Gradle as build system and creating an Eclipse editor.
  • mydsl.tests: The tests that only depend on the base infrastructure, i.e. not on any editor, should go here as plain JUnit tests. Generated when Testing Support is enabled.
  • mydsl.ui: The Eclipse editor plug-in, generated by the Eclipse plug-in facet.
  • mydsl.ui.tests: JUnit plug-in tests for the Eclipse editor plug-in go in here. Generated if Eclipse plug-in and Testing Support are checked.

The Workflow

The MWE2 workflow generated from the above wizard settings is shown below.

The new generator workflow uses a single new component of type XtextGenerator. If you are missing the former DirectoryCleaner and StandaloneSetup: They are now fields of the generator with good defaults.

The generator is configured with a DefaultGeneratorModule. It is a real Guice module, so if you need additional services in custom code generator fragments, you can use Xtext’s module API to bind them in a subclass and use that one instead. All other components can just @Inject any service. The DefaultGeneratorModule also binds an IXtextProjectConfig that defines what kind of Eclipse projects should be created and how they are named, and a CodeConfig holding common code generation parameters (see the source code for substituted variables in the file header).

The XtextGenerator also defines any number of IXtextGeneratorLanguages, each with a language name and a set of file extensions. The StandardLanguage defines a common set of generator fragments as fields, which can be customized individually as in the example. If you want to skip/add fragments, the easiest way is to copy StandardLanguage and adapt it.

By | 2016-11-30T08:34:08+00:00 February 4th, 2016|Eclipse, Generator, Xtext|12 Comments

About the Author:

Dr. Jan Köhnlein has been a passionate committer to Xtext and Xtend from the beginning. He is one of the leaders of TypeFox and drives the FXDiagram project.

12 Comments

  1. Adolfo SBH March 24, 2016 at 20:57 - Reply

    In case anybody ended up here with the same issue. We struggled to incorporate the Antlr fragment which allows to provide the ANTLR parser options. We concluded that we can’t add the fragment to the StandardLanguage because it gets ignored.

    Workaround: Copy the StandardLanguage and remove the contributions that you don’t like/want to replace. Then whatever fragment you add from the MWE works as expected.

    @Jan: Should not this be improved so that additions from the MWE replace the old ones? Am I missing anything ? I could send an enhancement request for that.

    • Adolfo SBH March 24, 2016 at 21:15 - Reply

      Never mind. I think I’ve found the correct way to deal with this.

      Rather than using fragments, the own StandardLanguage component allows you to deal with the ANTL parser options. Just do:

      parserGenerator = {
      options = { /*whatever options you need*/ }
      }

      Awesome !! Good work !!

  2. Suprith August 18, 2017 at 15:22 - Reply

    I am using this latest feature but I am confused in incorporating this into a maven build with my existing project. I want to generate files in a specific path with my existing project also I want to configure the grammarURI into the platform path. But I am stuck doing it. Can you please provide a relevant approach???

  3. Christoph November 9, 2017 at 18:21 - Reply

    I’m trying to write my own clone of StandardLanguage (in an Eclipse Plugin). It gives me some trouble as I can’t access the fragment classes such like GrammarAccessFragment2 due to Discouraged Access. I think this is related to the “x-internal” attribute on the fragment packages in the MANIFEST.MF in org.eclipse.xtext.xtext.generator plugin. Is there a workaround?

    • Jan Köhnlein November 10, 2017 at 09:26 - Reply

      We tend to mark new packages as internal in order to be able to fix bugs that pop up when clients are starting to use them without a lot of hassle.
      The new generator infrastructure has been around for quite a while now and we don’t expect any API changes. Writing your own StandardLanguage marks you as the kind of user that could safely ignore these warnings.

      • Christoph November 10, 2017 at 10:36 - Reply

        But how do I ignore these warnings (actual these are errors)? Eclipse is not giving me access to these classes.

        • Jan Köhnlein November 10, 2017 at 10:48 - Reply

          You can set the severity under Preferences > Java > Compiler > Errors/Warnings > Deprecated and restricted API > Discouraged reference (Access rules).

  4. Christoph November 10, 2017 at 10:53 - Reply

    Ok, thanks. Lets see how this translates to a tycho maven build…

    • Jan Köhnlein November 10, 2017 at 11:00 - Reply

      Usually, checking in the .settings does the job.

      • Christoph November 10, 2017 at 12:53 - Reply

        Works so far, but Junit4Fragment2 and SimpleProjectWizardFragment2 are not exported. Should be added to the Export-Package section of the MANIFEST.MT of org.eclipse.xtext.xtext.generator.

Leave A Comment