librelist archives

Parser not really eatting whitespace?

From:

Zachary Drummond - Bia

Date:

2011-08-24 @ 00:22

I followed the online docs pretty closely and got hit by the fact that the
parser does not seem to always eat whitespace. If you run the following 
code, you will see that the intergers have no whitespace, but the + does, 
which causes the transform to fail. Any ideas of what I am doing wrong?

Thanks,
-Zach
---
require 'parslet'

class MiniP < Parslet::Parser
  # Single character rules
  rule(:lparen) { str('(') >> space? }
  rule(:rparen) { str(')') >> space? }
  rule(:comma) { str(',') >> space? }

  rule(:space) { match('\s').repeat(1) }
  rule(:space?) { space.maybe }

  # Things
  rule(:integer) { match('[0-9]').repeat(1).as(:int) >> space? }
  rule(:identifier) { match['a-z'].repeat(1) }
  rule(:operator) { match('[+]') >> space? }

  # Grammar parts
  rule(:sum) { integer.as(:left) >> operator.as(:op) >> expression.as(:right) }
  rule(:arglist) { expression.as(:arg) >> (comma >> expression.as(:arg)).repeat }
  rule(:funcall) { identifier.as(:funcall) >> lparen >> 
arglist.as(:arglist) >> rparen }

  rule(:expression) { funcall | sum | integer }
  root :expression
end

class MiniT < Parslet::Transform
  rule(:int => simple(:x) ) { Integer(x) }
  rule(:op => '+', :left => simple(:l), :right => simple(:r)) { l + r }
end

def parse(str)
  mini = MiniP.new
  tree = mini.parse(str)

  print "Parsing #{str}: -> #{tree}\n"

  result = MiniT.new.apply( tree )
  print "Transform -> #{result}\n\n"

rescue Parslet::ParseFailed => error
  puts error, mini.root.error_tree
end

parse "1 + 2"
#Parsing 1 + 2: -> {:left=>{:int=>"1"@0}, :op=>"+ "@2, :right=>{:int=>"2"@4}}
#Transform -> {:left=>1, :op=>"+ "@2, :right=>2}

parse "1+2"
#Parsing 1+2: -> {:left=>{:int=>"1"@0}, :op=>"+"@1, :right=>{:int=>"2"@2}}
#Transform -> 3


Zachary Drummond
BIA | TotalDiscovery.com

2001 6th Ave
Seattle, WA 98040
O: 206-805-3131
www.biaprotect.com<http://www.biaprotect.com/>
www.TotalDiscovery.com<http://www.TotalDiscovery.com>


DEFENSIBLE DATA SOLUTIONS(tm)

______________________________________________________________________________________________________________________________________
This message may contain confidential and/or legally privileged 
information intended only for the use of the individual(s) named on the 
To:, Cc: and Bcc: lines.  If you are not one of the intended recipient(s),
then you are hereby notified that any disclosure, copying, distribution or
the taking of any action in reliance on the contents of this email 
transmission is strictly prohibited. If you have received this 
communication in error, please delete the email in its entirety and call 
or email us immediately so that we may take appropriate steps to correct 
the problem at no cost to you. Even if this message has reached you in 
error, sender does not in any way waive confidentiality or privilege. 
Thank you.

Zach,

The problem is that your operator rule is including the whitespace, so
:op => "+ " (note space) and thus your transform rule is not matching.

You should switch these two rules:

  rule(:operator) { match('[+]') >> space? }
  rule(:sum) { integer.as(:left) >> operator.as(:op) >> expression.as(:right) }

to this instead:

  rule(:operator) { match('[+]').as(:op) >> space? }
  rule(:sum) { integer.as(:left) >> operator >> expression.as(:right) }

and I believe that should work.

-mj

On Tue, Aug 23, 2011 at 8:22 PM, Zachary Drummond - BIA
<zdrummond@biaprotect.com> wrote:
> I followed the online docs pretty closely and got hit by the fact that the
> parser does not seem to always eat whitespace. If you run the following
> code, you will see that the intergers have no whitespace, but the + does,
> which causes the transform to fail. Any ideas of what I am doing wrong?
>
> Thanks,
>
> -Zach

Thanks! That did it. 

Note, the documentation on the site seems to be incorrect in a couple 
places. For example, I copied that operator code directly from the getting
started page

http://kschiess.github.com/parslet/get-started.html
<Example in the section " Making the parser complete"/>
class MiniP < Parslet::Parser
          # Single character rules
          rule(:lparen)     { str('(') >> space? }
          rule(:rparen)     { str(')') >> space? }
          rule(:comma)      { str(',') >> space? }
        
          rule(:space)      { match('\s').repeat(1) }
          rule(:space?)     { space.maybe }
        
          # Things
          rule(:integer)    { match('[0-9]').repeat(1).as(:int) >> space? }
          rule(:identifier) { match['a-z'].repeat(1) }
          rule(:operator)   { match('[+]') >> space? }
          
          # Grammar parts
          rule(:sum)        { integer.as(:left) >> operator.as(:op) >> 
expression.as(:right) }
          rule(:arglist)    { expression >> (comma >> expression).repeat }
          rule(:funcall)    { identifier.as(:funcall) >> lparen >> 
arglist.as(:arglist) >> rparen }
          
          rule(:expression) { funcall | sum | integer }
          root :expression
end

-----Original Message-----
From: ruby.parslet@librelist.com [mailto:ruby.parslet@librelist.com] On 
Behalf Of Melissa Whittington
Sent: Wednesday, August 24, 2011 8:42 AM
To: ruby.parslet@librelist.com
Subject: Re: [ruby.parslet] Parser not really eatting whitespace?

Zach,

The problem is that your operator rule is including the whitespace, so :op
=> "+ " (note space) and thus your transform rule is not matching.

You should switch these two rules:

  rule(:operator) { match('[+]') >> space? }
  rule(:sum) { integer.as(:left) >> operator.as(:op) >> expression.as(:right) }

to this instead:

  rule(:operator) { match('[+]').as(:op) >> space? }
  rule(:sum) { integer.as(:left) >> operator >> expression.as(:right) }

and I believe that should work.

-mj

On Tue, Aug 23, 2011 at 8:22 PM, Zachary Drummond - BIA 
<zdrummond@biaprotect.com> wrote:
> I followed the online docs pretty closely and got hit by the fact that 
> the parser does not seem to always eat whitespace. If you run the 
> following code, you will see that the intergers have no whitespace, 
> but the + does, which causes the transform to fail. Any ideas of what I 
am doing wrong?
>
> Thanks,
>
> -Zach