Hello Zach et all, I am working on a project where I plan on doing the following: * Represent the complete grammar of ECMAScript 3[0] in a code-as-data sense, similar to how Lispers use the term (possibly as a JSON grammar that Jison supports for use in CommonJS modules). * Create a parser from that grammar. * Write another module that walks the AST generated by the parser and outputs JavaScript code as a string. * Finally, provide helper functions/hooks to add new statements and expressions to the grammar *along with the code needed to transform that new statement/expression's AST to valid javascript*. Eventually, I want to use these parts to build a simple macro system JS similar to Meta Lua[1]. Is Jison the right tool for this job? If not, any suggestions for what might be? I looked in to Narcissus[2], but I do not think that it is extensible enough to handle my needs. I like that with Jison the grammars can be simple CommonJS modules which makes me think it wouldn't be too hard to write the code for providing new statements/expressions. Does this line of thinking make sense? Thanks for your time, _Nick_ [0] The ES3 grammar is available here http://www.mozilla.org/js/language/E262-3.pdf on page 159 [1] http://metalua.luaforge.net/ [2] http://mxr.mozilla.org/mozilla/source/js/narcissus/
Interesting ideas Nick. I'm using Jison for an ECMAScript parsing project[1] that uses the BNF notation of the grammar, but could trivially output the JSON version for easier modification from JavaScript code. It can also generate JavaScript source from the built up ASTs as well, and I am looking into adopting SpiderMonkey's API[2]. I currently don't have an elegant solution for extending grammars and your ideas seem to come closer. The tricky part I see is composing lexical grammars, which are sensitive to the order in which they are applied. Care must also be taken to ensure ambiguities don't arrise from composing grammar fragments. Perhaps these issues are not so great as to offset the potential features. Feel free to branch my work if you want (MIT licensed) or use it as a base. I plan to work on the SpiderMonkey API first, but your ideas are intriguing. [1] http://github.com/zaach/cafe [2] https://developer.mozilla.org/en/SpiderMonkey/Parser_API On Mon, Sep 6, 2010 at 4:28 PM, Nick Fitzgerald <fitzgen@gmail.com> wrote: > Hello Zach et all, > > I am working on a project where I plan on doing the following: > > * Represent the complete grammar of ECMAScript 3[0] in a code-as-data > sense, similar to how Lispers use the term (possibly as a JSON grammar that > Jison supports for use in CommonJS modules). > > * Create a parser from that grammar. > > * Write another module that walks the AST generated by the parser and > outputs JavaScript code as a string. > > * Finally, provide helper functions/hooks to add new statements and > expressions to the grammar *along with the code needed to transform that new > statement/expression's AST to valid javascript*. > > Eventually, I want to use these parts to build a simple macro system JS > similar to Meta Lua[1]. > > Is Jison the right tool for this job? If not, any suggestions for what > might be? I looked in to Narcissus[2], but I do not think that it is > extensible enough to handle my needs. I like that with Jison the grammars > can be simple CommonJS modules which makes me think it wouldn't be too hard > to write the code for providing new statements/expressions. Does this line > of thinking make sense? > > Thanks for your time, > > _Nick_ > > [0] The ES3 grammar is available here > http://www.mozilla.org/js/language/E262-3.pdf on page 159 > [1] http://metalua.luaforge.net/ > [2] http://mxr.mozilla.org/mozilla/source/js/narcissus/ > -- Zach Carter
I wrote a tiny little proof of concept of modifying grammars on the fly: http://gist.github.com/568682 It is admittedly very clunky (plus I'm not super familiar with working with parser generators like Jison and Bison, so it may be more clunky than needed). The problem is that I think it is too tied to the grammar rather than defining macros and extending the language. You are right that there is a danger of creating an ambiguous grammar, and if there is one, you might find out too late... The beauty of Lisp macros is that they don't need to change the parser/grammar! (Unless you want reader macros). Thats why this is so hard for JS. I might go the preprocessor route and just hack everything together in that... I'm going to look in to Cafe in a second here. It looks like you aren't using a CommonJS module for the grammar though, which was how I was hoping to create closures that could modify the grammar on the fly. Thanks for your reply, _Nick_ On Mon, Sep 6, 2010 at 4:31 PM, Zachary Carter <zack.carter@gmail.com>wrote: > Interesting ideas Nick. I'm using Jison for an ECMAScript parsing > project[1] that uses the BNF notation of the grammar, but could trivially > output the JSON version for easier modification from JavaScript code. > > It can also generate JavaScript source from the built up ASTs as well, and > I am looking into adopting SpiderMonkey's API[2]. > > I currently don't have an elegant solution for extending grammars and your > ideas seem to come closer. The tricky part I see is composing lexical > grammars, which are sensitive to the order in which they are applied. Care > must also be taken to ensure ambiguities don't arrise from composing grammar > fragments. > > Perhaps these issues are not so great as to offset the potential features. > Feel free to branch my work if you want (MIT licensed) or use it as a base. > I plan to work on the SpiderMonkey API first, but your ideas are intriguing. > > > [1] http://github.com/zaach/cafe > [2] https://developer.mozilla.org/en/SpiderMonkey/Parser_API > > > On Mon, Sep 6, 2010 at 4:28 PM, Nick Fitzgerald <fitzgen@gmail.com> wrote: > >> Hello Zach et all, >> >> I am working on a project where I plan on doing the following: >> >> * Represent the complete grammar of ECMAScript 3[0] in a code-as-data >> sense, similar to how Lispers use the term (possibly as a JSON grammar that >> Jison supports for use in CommonJS modules). >> >> * Create a parser from that grammar. >> >> * Write another module that walks the AST generated by the parser and >> outputs JavaScript code as a string. >> >> * Finally, provide helper functions/hooks to add new statements and >> expressions to the grammar *along with the code needed to transform that new >> statement/expression's AST to valid javascript*. >> >> Eventually, I want to use these parts to build a simple macro system JS >> similar to Meta Lua[1]. >> >> Is Jison the right tool for this job? If not, any suggestions for what >> might be? I looked in to Narcissus[2], but I do not think that it is >> extensible enough to handle my needs. I like that with Jison the grammars >> can be simple CommonJS modules which makes me think it wouldn't be too hard >> to write the code for providing new statements/expressions. Does this line >> of thinking make sense? >> >> Thanks for your time, >> >> _Nick_ >> >> [0] The ES3 grammar is available here >> http://www.mozilla.org/js/language/E262-3.pdf on page 159 >> [1] http://metalua.luaforge.net/ >> [2] http://mxr.mozilla.org/mozilla/source/js/narcissus/ >> > > > > -- > Zach Carter >
On Tue, Sep 7, 2010 at 1:34 PM, Nick Fitzgerald <fitzgen@gmail.com> wrote: > I wrote a tiny little proof of concept of modifying grammars on the fly: > http://gist.github.com/568682 It is admittedly very clunky (plus I'm not > super familiar with working with parser generators like Jison and Bison, so > it may be more clunky than needed). The problem is that I think it is too > tied to the grammar rather than defining macros and extending the language. > You are right that there is a danger of creating an ambiguous grammar, and > if there is one, you might find out too late... The beauty of Lisp macros is > that they don't need to change the parser/grammar! (Unless you want reader > macros). Thats why this is so hard for JS. I might go the preprocessor route > and just hack everything together in that... > Yeah, Lisp is basically an AST, so it's easier. > > I'm going to look in to Cafe in a second here. It looks like you aren't > using a CommonJS module for the grammar though, which was how I was hoping > to create closures that could modify the grammar on the fly. > > Right, I'd have to convert the grammar to JSON so it could be included in a CommonJS module, which is doable. > Thanks for your reply, > > _Nick_ > > > > > On Mon, Sep 6, 2010 at 4:31 PM, Zachary Carter <zack.carter@gmail.com>wrote: > >> Interesting ideas Nick. I'm using Jison for an ECMAScript parsing >> project[1] that uses the BNF notation of the grammar, but could trivially >> output the JSON version for easier modification from JavaScript code. >> >> It can also generate JavaScript source from the built up ASTs as well, and >> I am looking into adopting SpiderMonkey's API[2]. >> >> I currently don't have an elegant solution for extending grammars and your >> ideas seem to come closer. The tricky part I see is composing lexical >> grammars, which are sensitive to the order in which they are applied. Care >> must also be taken to ensure ambiguities don't arrise from composing grammar >> fragments. >> >> Perhaps these issues are not so great as to offset the potential features. >> Feel free to branch my work if you want (MIT licensed) or use it as a base. >> I plan to work on the SpiderMonkey API first, but your ideas are intriguing. >> >> >> [1] http://github.com/zaach/cafe >> [2] https://developer.mozilla.org/en/SpiderMonkey/Parser_API >> >> >> On Mon, Sep 6, 2010 at 4:28 PM, Nick Fitzgerald <fitzgen@gmail.com>wrote: >> >>> Hello Zach et all, >>> >>> I am working on a project where I plan on doing the following: >>> >>> * Represent the complete grammar of ECMAScript 3[0] in a code-as-data >>> sense, similar to how Lispers use the term (possibly as a JSON grammar that >>> Jison supports for use in CommonJS modules). >>> >>> * Create a parser from that grammar. >>> >>> * Write another module that walks the AST generated by the parser and >>> outputs JavaScript code as a string. >>> >>> * Finally, provide helper functions/hooks to add new statements and >>> expressions to the grammar *along with the code needed to transform that new >>> statement/expression's AST to valid javascript*. >>> >>> Eventually, I want to use these parts to build a simple macro system JS >>> similar to Meta Lua[1]. >>> >>> Is Jison the right tool for this job? If not, any suggestions for what >>> might be? I looked in to Narcissus[2], but I do not think that it is >>> extensible enough to handle my needs. I like that with Jison the grammars >>> can be simple CommonJS modules which makes me think it wouldn't be too hard >>> to write the code for providing new statements/expressions. Does this line >>> of thinking make sense? >>> >>> Thanks for your time, >>> >>> _Nick_ >>> >>> [0] The ES3 grammar is available here >>> http://www.mozilla.org/js/language/E262-3.pdf on page 159 >>> [1] http://metalua.luaforge.net/ >>> [2] http://mxr.mozilla.org/mozilla/source/js/narcissus/ >>> >> >> >> >> -- >> Zach Carter >> > > -- Zach Carter