Xtext 2.13.0 released: semantic editing made easy

Today we are excited to release Xtext 2.13.0 and I would like to give you an opinionated tour through some of the new features. There is one area where particularly much happened: Refactoring support and QuickFix support. I want to thank Jan and Dennis for implementing significant parts of it. Refactoring and QuickFix are similar: They take the DSL documents and turn them into some improved version of themselves. While refactoring focusses on restructuring, QuickFixes focus on resolving problems. The modifications applied to the DSL documents can range from very simple cases like updating an attribute’s value to complex cases such as renaming or moving multiple elements at once. To implement an operation like this, in the past, it was either necessary to obtain tokens from the node model and directly modify the text or obtain the AST and serialize the modified subtree. Unfortunately, both approaches have drawbacks:

• Changing text directly, even with the help of the node model, is only practical for small changes and insertions. Your language’s formatter is not used and syntactic correctness of the change is your very own responsibility.
• Neither approach can detect and update cross references that point to the modified piece of the text or AST.
• The serialization approach is too fragile to work with broken models since it needs to re-serialize the full sub-tree of the modified element. Unfortunately, since we’re working with text that’s being edited by a human, broken models are more common than valid ones.
• And more.

ChangeSerializer: engine for semantic editing

So we spent lots of thought on how Xtext could provide hooks where all you need to do is to update your AST (the EMF model) and Xtext will take care of updating the textual DSL documents. The result is an amazingly simple service:

public interface IChangeSerializer {

  interface IModification<T extends Notifier> {
      void modify(T context);
  }

  <T extends Notifier> void addModification(T context, IModification<T> modification);
  void applyModifications(IAcceptor<IEmfResourceChange> acceptor);
  void setUpdateCrossReferences(boolean value);
  void setUpdateRelatedFiles(boolean value);

  // (...)
}

To use this service, first, you call addModification(context, modification) one or multiple times. The parameter context is the EMF Resource or EObject that (children included) you want to modify. The parameter modification is an object or a lambda that executes the modification. When all modification have been added, it is time to call applyModifications(acceptor). With acceptor, you provide a call-back that will be called once for every EMF Resource that needs to be changed. For XtextResources, you’ll get instances of

public interface ITextDocumentChange extends IEmfResourceChange {
  URI getOldURI();
  URI getNewURI();
  XtextResource getResource();
  List<ITextReplacement> getReplacements();
}

As promised, this gives all ITextReplacements that are needed to apply the model change back to the textual DSL documents. A lot is being considered during this process:

• The changes are kept as minimal as possible. For example, when only the value of a single EAttribute is changed, only the EAttributes text region will be updated. This keeps serialization robust in times of broken models. To add new EObjects or to handle very complicated changes, the ChangeSerializer will delegate to ISerializer .
• Modifications can occur in one or many files at the same time. This feature is important because often files are interconnected via cross references and therefore it can be impossible to treat the individually.
• XtextResources and other EMF resources can be mixed. For example, XML or XMI resources can be involved.
• Cross references are updated automatically when setUpdateCrossReferences(true) has been set. We built in this feature because we realized that more advanced modifications often involve many rename-refactorings.
• Related files are automatically detected and updated when setUpdateRelatedFiles(true) has been called. Related files are the ones for which there is no explicit modification, but which contain cross references that need to be updated. Related files are determined using Xtext’s index.
• There is a hook to update the model of related files, for example to update a section of import statements.
• Code comments are handled properly by being moved/deleted according to how the associated model elements are moved/deleted.
• Transactional model modification: Since all changes are computed before they are applied, a modification can be aborted at any time withour having caused side-effects.
• Your IFormatter2 is called for all model modifications. Thus, the resulting text changes are always nicely formatted according to how you implemented your formatter.

The IChangeSerializer is part of xtext.ide and therefore available on all platforms supported by Xtext: Language Server Protocol, Eclipse IDE, etc.

File/folder copy/move/rename refactoring

In short: Resource Relocation Refactoring. The motivation behind this refactoring is the fact that many languages support syntactical elements that directly relate to the structure of the file system:

• Java-style package names: By definition symmetric to the file’s path within the source folder.
• C-style import statements: They’re like cross references pointing to file names.

If the file structure is changed by renaming files or folders or by moving them around, the contents of one or more files needs to be updated. A perfect case for the change serializer! Also, when a file is copied it must have a different path/name compared to the original, thus making file contents changes necessary. Implementing such a hook is spectacularly simple:

class MyStrategy implements IResourceRelocationStrategy {

  @Inject IResourceServiceProvider language

  override applyChange(ResourceRelocationContext context) {
    context.changes.filter[language.canHandle(fromURI)].forEach [ change |
      context.addModification(change) [ resource |
        val rootElement = resource.contents.head
        if (rootElement instanceof PackageDeclaration) {
          val newPackage = change.toURI.trimSegments(1).segmentsList.drop(2).join('.')
          rootElement.name = newPackage
        }
      ]
    ]
  }
}

Even though the hook’s implementation is language-specific, it receives the URIs of all changed files, independently of which language they belong to. This enables you to implement piggyback renaming, e.g. rename a diagram file when its model file has been renamed. Therefore, the first thing this example does is filter out all URIs of other languages via canHandle. Then it creates a modification lambda in which the AST is accessible and it simply updates the EAttibute name with the new package name. The ChangeSerializer will automatically detect which other files are referencing these files and update those accorrdingly.

Multi-quickfixes

We also made available a new QuickFix API based on the ChangeSerializer. It has interesing capabilities:

• Multi-QuickFixes are now supported by Xtext: As a user you can select multiple markers from the Eclipse Problems View and fix them all at once with a single action. It’s hard to over-state how awesome this is!
• Cross-File QuickFixes: A single QuickFix can now update several files at once.
• File rename/move simply by changing the URI in the EMF resource.
• Automatic updating of affected cross references: Changing a model element’s name can now be a rename refactoring!

@Fix(MyValidator.ISSUE_CODE_1)
public void fix1(final Issue issue, IssueResolutionAcceptor acceptor) {
  acceptor.acceptMulti(issue, "Add Doc1", "Adds Documentation", null,
    (MyElement myEObject) -> {
      myEObject.setDoc("Better documentation");
    }
  );
}

@Fix(MyValidator.ISSUE_CODE_2)
public void fix2(final Issue issue, IssueResolutionAcceptor acceptor) {
  acceptor.acceptMulti(issue, "Add Doc2", "Adds Documentation", null,
    (MyElement myEObject, ICompositeModificationContext<MyElement> context) -> {
      ctx.addModification(main, (obj) -> {
        obj.setDoc("Better documentation");
      }
    );
  });
}

The API works just like the old one, except that that now you’ll need to call acceptor.acceptMulti() instead of acceptor.accept(). There are two flavors of the API:

• The one with one parameter: myEObject : In the lambda, you may modify any EObject from the same resource. Cross references and related files will be updated.
• The one with two parameter: myEObject and context . The context object gives you fine-grained control over the ChangeSerializer which is operating behind the scenes. AST-modifications are only allowed via ctx.addModification() . Since you tell ctx.addModification about the root element about the modification, elements from other resources can be modified and/or you can keep the recorded model-subtree as small as possible. Also, via the context object, you can enable/disable updating of cross references and related files.

The second QuickFix enables a pattern that’s important for Multi-Fixes: First resolve everything, then modify. For example when you’re storing EObject URIs in an issue’s user data, you’ll need to resolve them before the first modification is applied. Otherwise, URIs may become unresolvable because the model has changed. These two stages are automatically handled properly when you do the resolution outside of ctx.addModification() and the actual modification inside ctx.addModification’s lambda.

Rename element refactoring

Rename refactoring for model elements has long been available in Eclipse Xtext. New in Xtext 2.13.0 is a flag to enable the ChangeSerializer to be the driving engine:

Workflow {
  component = XtextGenerator {
    ...
    language = StandardLanguage {
      ...
      renameRefactoring = {
        useChangeSerializer = true
      ...

And more!

Among the other changes in Xtext 2.13 there is:

• Tons of bug-fixes, discussion, reviews. Thank you to all committers and Christian in particular.
• The project creation wizard can now create projects that contain build scripts that assemble regular or fat JARs for Language Servers. Thx Karsten!
• Improvements for indentation-based languages. Thank you Sebastian!
• New collection literals for empty collections and Iterables.flatMap() for xbase.lib (Thx Karsten!)