TinyMCE 4 Spellchecker integration


TinyMCE is on it’s way to release new major update to 4.0. You may want to see a demo for a quick highlights of updates.

Well, I just realized that I’m writing this post in TinyMCE 3.5.8. 4.0 it is still in beta stage but probably we’ll see the upgrades to 4.0 in a few months after it will have final release.

Anyway, while doing updates to JSpellChecker (see below for a list) and developing the “usage example” project, I downloaded latest TinyMCE 4.0 release and was surprised by the error’s I see after trying to perform spellcheck. The thing is that TinyMCE 4.0 spellchecker plugin has a lot of updates and actually now it’s using completely different request structure than before (in 3.x). There is no docs yet which document the new spellchecker request structure, so I did investigation of requests in java debug and in Chrome developer console.

The good news about new spellchecker API is that it seems to be more optimal, since it execute only one request in which it sends words he want to check and expect the suggestions back (for all words) for misspelled words. So, user who use editor will be able to see suggestions right after he got response. Not sure how that will impact the server-side of spellchecker, but generally I think that algorithms could be optimized and suggestions to all words could be retrieved faster (since dictionaries/indexes are already loaded at the moment we look for misspelled words). At least number of potential requests to server will be much lower.

The bad news (on the moment) is that only one language is supported on the moment (spellchecker plugin always pass “lang”=”en”). There is a bug filled and I believe it will be fixed before final release. But you are welcome to vote for it!

TinyMCE site does not provide documentation on request/response structure for the new spellchecker API (actually it does not provide it for previous version too). Information below is a result of reverse engineering of new spellchecker plugin code.
Well, so the request structure is updated and now only one request is done by spellchecker plugin.
See example spellchecker request/response comparison table below:

Before TinyMCE 4.0 Since TinyMCE 4.0

“checkWords” request

Check the list of words passed as a second argument of the “params” array. First argument of the params array, define the “language”

Example request
{
"method" : "checkWords",
"params" : [ "en-us", ["usa","cawboy", "apple"] ]
}
Example Response
"id":null,
"result":["cawboy"],
"error":null}

“spellcheck” request

It is done by a single request/response now. “params” is now JsonObject not JsonArray and that make request more structured

Example request
{
"id"  : ${request-id},
"method" : "spellcheck",
"params" : {
 "lang": "en-us", 
 "words": ["usa","cawboy", "apple", "blablabla"] 
}
}
Example Response

The “result” will contain mappings where key is misspelled word and value is suggestion list for it. It also may return empty suggestion list for a word and that will mark it as “misspelled”

{
"id":"${request-id}",
"result": {
  "cawboy" : ["cowboy"],
  "Guugli" : ["Google"],
  "blablabla" : []
 }
}

“getWordSuggestions” request

Get suggestion for single word, passed as the 2-nd element in the “params” array

Example request
{
"method" : "getWordSuggestions",
"params" : [ "en-us", "Guugli" ]
}
Example Response
{"id":null,
"result":["Google"],
"error":null
}

Btw, the JspellChecker code is updated to support TinyMCE 4.0 request/response format. It also contains other refactorings, updates and fixes which I will cover in the next post

Advertisements

https://www.facebook.com/achorniy

Tagged with: , , , , , , , , , , , , , ,
Posted in Software Development
21 comments on “TinyMCE 4 Spellchecker integration
  1. superaldo says:

    Where can I download JSpellChecker ? It is not already available

  2. superaldo says:

    Sorry finally, I found it. But it is very difficult to implement it, do you have any tuto ?

  3. Mike Yudin says:

    Hi Andrey,

    Since you are one of the few resources on the net who have implemented jazzy with TinyMCE >=4.0, I’m coming to you with a question/hoping for feedback:

    I have implemented a similar solution in one of our projects (Instead of using plain servlets, I am using Struts2 to serve up jazzy suggestions — might switch to servlets). Additionally, I have created a forked version of the TinyMCE spellchecker plugin to do as-you-type spellchecking (every time they hit the spacebar or lose focus on tinymce).

    My reason for this is I feel our end users will not take the time to click the spellcheck button. My question to you is: How do you feel about this solution? Obviously this results in a lot of calls to the server(exponentially based on the number of users we have). We have the dictionary loaded into memory, so I’m wondering if that would keep the memory usage intact.

    Thanks for your contributions!

    -Mike

    • Andrey Chorniy says:

      Hi Mike, yep, keeping all dictionaries in memory is good,
      I think it might be the real problem if the number of users will be large enough.
      And you should put your efforts to make sure that number of spellcheck requests per-second will not exceed some limit (hard to say which limit, but it could be checked in practice)

      First of all I would like to suggest you to update your code of spellhecker plugin, to limit maximum frequency of requests (so it will not send 3 requests per second even if events to send request are fired, but for example 1 request per 10 seconds). You may just calculate “nextRequestTime” once you will receive request and once you will detect you need to do spellcheck – scheule timet to (nextRequestTime)

      Assuming that you implemented your own plugin for spellchecker you may also add additional parameter from server, which will control the client max-frequency. That will allow you to keep max-frequency of requests which received by server under some level. So, along with spelcheck response you may pass back the new param like “nextRequestDelay” and use it on client side. You will need to calculate it on server depending on number of requests per minute.

      That would be also good to send not all text to server, but only new words (to not recheck everything each time) – kind of “incremental spellcheck”, but that might require some serious updates in spellchecker plugin. What you need here is “collect” new words since last spellcheck and send only those words (plus do not remove previous spellcheck results)

      • Andrey Chorniy says:

        In addition to all of that, you may use something like RequestRateThrootleFilter for your spellcheck requests, just to be calm and sure that your server will not be overloaded 🙂

        I use this implemntation from owasp-esapi before
        https://code.google.com/p/owasp-esapi-java/source/browse/trunk/src/main/java/org/owasp/esapi/filters/RequestRateThrottleFilter.java

      • Mike Yudin says:

        Andrey,

        Great ideas. Thank you. I have implemented this solution, so that we don’t do a request more than once every 2 seconds. I might also consider only doing the spellcheck when they enter punctuation (period, exclamation, etc) and also when they lose focus of the TinyMCE element.

        I have also implemented the RequestRateThrottleFilter to limit the number of requests.
        .
        Since this application currently won’t be used by a ton of users, I’m not too worried about overloading the server with requests (the server also has something like 80GB of ram), but it’s best to put a solution into place before anything gets out of hand.

        Thanks for your help!

        -Mike

  4. Carla says:

    Andry you are awesome!! Worked like a charm.The explanation is super simple. :)Thanks a lot!

  5. RV says:

    Hi I am facing issue while running jspellchecker with TinyMCE 4.0.23 . I have created below topic. Any help appreciated.

    http://www.tinymce.com/forum/viewtopic.php?id=33816

  6. Have you tried to download latest code “svn checkout svn://svn.code.sf.net/p/jspellchecker/code/trunk jspellchecker-code” and try to deploy jspellchecke and example app ?

    • RV says:

      Thanks for reply ,

      I downloaded revision 10 from Branch and created project in eclipse. Created jspellcheck.war and replaced exisiting war file. and then used configured tinyMCE to use that.

      plugins : “spellchecker paste charmap link”,
      spellchecker_languages : “+English=en”,
      spellchecker_rpc_url : “/jspellchecker/jazzy-spellchecker”,
      toolbar : ” spellchecker charmap”,

      And still i got error.

      Do i nee to change anything in TinymCE core files.

      As suggested by you i can try to deploy example app and then let you know my findings.

      Thanks!

  7. Mike says:

    Andrey,

    Where can I find the dictionaries to download?

  8. David C says:

    This has changed again. The next expected Request looks like:
    {“method”=>”spellcheck”, “text”=>”xxx acidifyuiu”, “lang”=>”en”}

    Expected response should look like:

    {
    :id => nil,
    :words => {
    “xxx” => [
    [0] “xx”,
    [1] “xix”,
    [2] “x xx”,
    [3] “xx x”,
    [4] “XXL”,
    [5] “XOR”,
    [6] “XML”,
    [7] “X’s”,
    [8] “Sax”,
    [9] “Say”
    ],
    “acidifyuiu” => [
    [0] “acidify”,
    [1] “acidifiable”,
    [2] “acidification”,
    [3] “acidimetry”,
    [4] “mystifying”,
    [5] “acquisitive”
    ]
    },
    :error => nil
    }

    See (method: defaultSpellcheckCallback): https://github.com/tinymce/tinymce/blob/master/js/tinymce/plugins/spellchecker/classes/Plugin.js

  9. David C says:

    And that is using the latest TinyMce build.

    • Ups, which version of TinyMCE it is ? is this version already released ?

      • yunpeng says:

        Hi Andrey,

        TinyMCE latest version is 4.2.5 now. The request and JSON response format have changed again. See http://www.tinymce.com/wiki.php/Configuration:spellchecker_rpc_url

        I read your source code of TinyMCESpellCheckerServlet and redo the doPost() as the following:

        protected void doPost(HttpServletRequest request, HttpServletResponse response)
        throws ServletException, IOException {

        setResponeHeaders(response);

        try {

        //——————————————————————————————-
        //1. Retrieve values of the request parameters (method, text, lang).
        //——————————————————————————————-
        String methodName = request.getParameter(“method”);
        String text = request.getParameter(“text”);
        String lang = request.getParameter(“lang”);

        //TODO
        System.out.println(“methodName=”+methodName+”, lang=”+lang);

        //————————————————–
        //2. Construct a list of words for spell checking.
        //————————————————–
        List inputWords = constructWords(text);

        JSONObject jsonOutput = new JSONObject();
        SpellcheckMethod method = SpellcheckMethod.valueOf(methodName.trim());

        switch (method) {

        case spellcheck:
        //——————————————————————————-
        //3. For each word, do a spell-check. If a word is misspelled, add suggestions.
        //——————————————————————————-
        List misspelledWordsList = findMisspelledWords(inputWords.iterator(), lang);

        JSONObject misspelledWordJSON = new JSONObject();

        for(String misspelled : misspelledWordsList){

        JSONArray suggestionJSONArray = getWordSuggestions(misspelled, lang);

        misspelledWordJSON.put(misspelled, suggestionJSONArray);

        }

        //————————————————–
        //4. Construct JSonObject in the correct format.
        //————————————————–
        jsonOutput.put(“words”, misspelledWordJSON);

        //TODO
        System.out.println(jsonOutput);

        break;
        default:
        throw new SpellCheckException(“Spellchecker method not supported {” + methodName + “}”);
        }

        PrintWriter pw = response.getWriter();
        pw.println(jsonOutput.toString());

        } catch (SpellCheckException se) {
        logger.warn(“Failed to perform spellcheck operation”, se);
        returnError(response, se.getMessage());
        } catch (Exception e) {
        logger.warn(“Failed to perform spellcheck operation (generic error)”, e);
        returnError(response, e.getMessage());
        }

        response.getWriter().flush();
        }

        Also, I added and modified these methods:

        private List constructWords(String text) {
        List words = new ArrayList();

        if(text != null && text.trim() != null){
        Pattern pattern = Pattern.compile(“\\w+”);
        Matcher matcher = pattern.matcher(text.trim());
        while (matcher.find()) {
        String word = matcher.group();

        //TODO
        System.out.println(word);

        words.add(word);
        }
        }
        return words;
        }

        private JSONArray getWordSuggestions(String word, String lang) throws SpellCheckException {
        JSONArray suggestions = new JSONArray();
        if (word != null) {

        List suggestionsList = findSuggestions(word, lang, maxSuggestionsCount);
        for (String suggestion : suggestionsList) {
        suggestions.put(suggestion);
        }
        }
        return suggestions;
        }

        Then it seems work again now.

      • Great, thanks, it probably worth to switch to github or bitbucket, so it could be easily collaborated.
        I will take a look at updates this week bit later

  10. First step done, code (trunk) has been moved to github
    https://github.com/andrey-chorniy/jspellchecker
    will add your changes soon

  11. Hi Yunpeng
    Sorry, it takes long (have no time to do updates)
    Your updates are in this branch, please let me know if they works, feel free to suggest updates
    https://github.com/andrey-chorniy/jspellchecker/tree/TinyMCE-4.2.5

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: