librelist archives

« back to archive

Autorelative references and Liberty parser

Autorelative references and Liberty parser

From:
Paolo Redaelli
Date:
2011-10-21 @ 12:13
Following my obsession to build a "fully distributed" compiler 
infrastructure I'm trying to implement autorelative references as 
described in 
http://en.wikipedia.org/wiki/Pointer_%28computing%29#Autorelative_pointer
Since it may require some little changes in the runtime (adding some 
features to POINTER) I'm going to create a little branch.
Autorelative references (from here arr like the name of the branch I'm 
going to create in my repository) are useful to build relocatable objects.
Let's say the various processes will communicate using zeromq; if you 
don't care about performace every message can be XML or JSON or BSON but 
all those require building the message on the sender and parsing on the 
receiver.
To get fullest perfomance we may send a memory region containing Eiffel 
objects as a message. To achieve this we have to comply to two 
requirements. First, all references between the objects contained in a 
message to be autorelative references, otherwise the receiver should 
adjust all them; surely accessing autorelative references has some 
performance penalty: I will benchmark it a little.
Second a message should not contain plain references and or autorelative 
references that points "outside" the message.
In the first tentative design I had in mind all processes - even remote 
- will be spawned by a "central autority" (the "compiler command" issued 
by developer) so messages may be trusted. Endianness is a issue that can 
be handled by a proper heir of ømq messages.

I tried to "simplify" SmartEiffel parser to spawn several instances of 
it and I found its design monolitic as Uluru 
(http://en.wikipedia.org/wiki/Uluru) so I tried to reuse Liberty parser; 
I found myself at home.
Very well done, Adrian!
Please do not delete that parser as you wrote some time ago.

Re: [libertyeiffel] Autorelative references and Liberty parser

From:
Cyril Adrian
Date:
2011-10-22 @ 08:00
Hi Paolo,

2011/10/21 Paolo Redaelli <paolo.redaelli@gmail.com>
>
> In the first tentative design I had in mind all processes - even remote
> - will be spawned by a "central autority" (the "compiler command" issued
> by developer) so messages may be trusted. Endianness is a issue that can
> be handled by a proper heir of ømq messages.
>

Endianness is not the only problem if you have an heterogeneous park of
machines. In that case, better stick with a fully defined protocol (JSON,
CORBA or whatever).
I'm not even sure the gain intra-machine is worth the work. You'd end up
with a SE-centric system with no way to send messages to other systems.

I'd rather implement all known protocols and be done.


> I tried to "simplify" SmartEiffel parser to spawn several instances of
> it and I found its design monolitic as Uluru
> (http://en.wikipedia.org/wiki/Uluru)


:-)

Why do you need more instances?


> so I tried to reuse Liberty parser;
> I found myself at home.
> Very well done, Adrian!
>

… Cyril (Adrian is my family name)

but thanks :-)

Please do not delete that parser as you wrote some time ago.
>

I wrote about deleting useless code such as the Liberty interpreter and
such. The Liberty parser will remain, because it is or will be used by other
tools.

Anyway I won't do it now because I'm not finished with the SmartEiffel
interpreter.

Also remember we use git and everything "deleted" can be called back from
the dead :-)

Cheers,

*Cyril ADRIAN*
*http://www.cadrian.net/cyril*
[image: Facebook] <https://www.facebook.com/cyril.adrian> [image:
Twitter]<https://twitter.com/cadbart> [image:
LinkedIn] <http://fr.linkedin.com/in/cadrian> [image: Google
Plus]<https://plus.google.com/100388810006463519079/posts> [image:
Google 
Calendar]<https://www.google.com/calendar/embed?src=1t93vvvrdc26ee0f83p0cunj60%40group.calendar.google.com&ctz=Europe/Paris>
Contact me: [image: Google Talk] cyril.adrian@gmail.com
 [image: Twitter] <http://twitter.com/cadbart> Latest tweet: [Liberty]
http://t.co/HpiAgTb5 Cyril Adrian - once strings!
Follow @cadbart <http://twitter.com/cadbart> Reply

<http://twitter.com/?status=@cadbart%20&in_reply_to_status_id=127143290729529340&in_reply_to=cadbart>
Retweet

<http://twitter.com/?status=RT%20%40cadbart%3A%20%5BLiberty%5D%20http%3A%2F%2Ft.co%2FHpiAgTb5%20Cyril%20Adrian%20-%20once%20strings!>
 00:04 Oct-21<http://twitter.com/cadbart/statuses/127143290729529344>
  Get this email app!

<http://www.wisestamp.com/apps/twitter?utm_source=extension&utm_medium=email&utm_term=twitter&utm_campaign=apps>

 [image: Google Plus] <http://plus.google.com/100388810006463519079> My
latest G+: Dizzying but invisible depthYou just went to the Google home
page.... <http://plus.google.com/100388810006463519079/posts/RDiBNaX2XA7/>
My G+ <http://plus.google.com/100388810006463519079> -
Posts<http://plus.google.com/100388810006463519079/posts/>- Add
to Circles <http://plus.google.com/100388810006463519079/about/> - 09:32
Oct-17 <http://plus.google.com/100388810006463519079/about/>
  Get this email app!

<http://www.wisestamp.com/apps/plus?utm_source=extension&utm_medium=email&utm_term=plus&utm_campaign=apps>

[image: My QR VCard]
  Get a signature like this.

<http://r1.wisestamp.com/r/landing?promo=17&dest=http%3A%2F%2Fwww.wisestamp.com%2Femail-install%3Futm_source%3Dextension%26utm_medium%3Demail%26utm_campaign%3Dpromo_17>
CLICK

HERE.<http://r1.wisestamp.com/r/landing?promo=17&dest=http%3A%2F%2Fwww.wisestamp.com%2Femail-install%3Futm_source%3Dextension%26utm_medium%3Demail%26utm_campaign%3Dpromo_17>

Re: [libertyeiffel] Autorelative references and Liberty parser

From:
Paolo Redaelli
Date:
2011-10-22 @ 23:01
Il 22/10/2011 10:00, Cyril ADRIAN ha scritto:
> Hi Paolo,
>
> 2011/10/21 Paolo Redaelli <paolo.redaelli@gmail.com
> <mailto:paolo.redaelli@gmail.com>>
>
>     In the first tentative design I had in mind all processes - even
>     remote
>     - will be spawned by a "central autority" (the "compiler command"
>     issued
>     by developer) so messages may be trusted. Endianness is a issue
>     that can
>     be handled by a proper heir of ømq messages.
>
>
> Endianness is not the only problem if you have an heterogeneous park
> of machines. In that case, better stick with a fully defined protocol
> (JSON, CORBA or whatever).
> I'm not even sure the gain intra-machine is worth the work. You'd end
> up with a SE-centric system with no way to send messages to other systems.
My "even remote" was a little misleading; of course inter-machine
communication shall be made using clear-text Json following the Unix
tradition (The Art of Unix programming is quite clear on that); at most
we may use Bson.
The issue is what to use on current high-end machines which are beasts
able to run several dozen threads/task concurrently. To gain the most
from those architecture I think we shall have a zero-copy IPC mechanism.
Building a Json message and sending it as a message is not such a
mechanism (my AmigaOS heritage shows here :-)
I searched for a way to memory map the same memory region at the same
virtual address in different processes in order to directly place an
ordinary hierarchy of Eiffel objects but it seems that there is no
portable way; I still wonder if it is even feasible; a tentative
solution is initialize a memory mapped region and populate it with
Eiffel objects that have auto-relative references instead of plain
references (i.e. foo: AUTORELATIVE_REFERENCE[FOO] instead of foo: FOO).
My first idea was to use ømq with several thread; in that case ømq uses
a real zero-copy approach when messages, so when SE runtime will be
somehow thread-safe we may also get rid of the mmap.

> I'd rather implement all known protocols and be done.
>  
>
>     I tried to "simplify" SmartEiffel parser to spawn several instances of
>     it and I found its design monolitic as Uluru
>     (http://en.wikipedia.org/wiki/Uluru)
>
>
> :-)
>
> Why do you need more instances?
To run 6 times faster on an Core i7 and 12 times on dual Xeons than
current SE.... :)
Seriously parallelizing the parsing is just a gym to start parallelizing
the rest... See http://en.wikipedia.org/wiki/Multi-core_processor x64
already has 8-core CPU, ARM 4, several startups have made CPUs with 100
or even 4096 cores... Tilera's 64/100 cores CPU is already supported by
Linux...
Liberty/SmartEiffel just cannot afford to remain a single-core
language/application.
When dealing with multi-threading ømq approach fits very well with
Eiffel philosophy: threads never exchange data, they send each other
messages; just add the requirement that the sender owns the message and
that it will not release it until the receiver answered it (as soon as
possible) and you get no memory issues at all.
>
>     so I tried to reuse Liberty parser;
>     I found myself at home.
>     Very well done, Adrian!
>
>
> … Cyril (Adrian is my family name)
>
> but thanks :-)
Oh, pardon, Adrian is used as a first name here...
> Also remember we use git and everything "deleted" can be called back
> from the dead :-)
I know I know :-)

Re: [libertyeiffel] Autorelative references and Liberty parser

From:
Cyril Adrian
Date:
2011-10-24 @ 09:17
2011/10/23 Paolo Redaelli <paolo.redaelli@gmail.com>
>
>  Why do you need more instances?
>
> To run 6 times faster on an Core i7 and 12 times on dual Xeons than current
> SE.... :)
>

Parallelizing the core of compile_to_c is a big work.
Don't start with the parser, I think it's not worth it (it is really
efficient and takes a really small portion of overall time).

Better try to distribute tasks by adding synchronization points to
specialize_in, specialize_thru, specialize_2.
Look at FEATURE_ACCUMULATOR, ANONYMOUS_FEATURE_MIXER. Those two classes are
the actual heart and liver of SmartEiffel.

Another possibility is the C_PRETTY_PRINTER which generates the C slices
(happily the C generation now fully lives in src/smarteiffel/generation/c).


 Liberty/SmartEiffel just cannot afford to remain a single-core
> language/application.
>

It is not: it already supports parallelizing the compilation of C slices and
it's already a big boost :-)


 When dealing with multi-threading ømq approach fits very well with Eiffel
> philosophy: threads never exchange data, they send each other messages; just
> add the requirement that the sender owns the message and that it will not
> release it until the receiver answered it (as soon as possible) and you get
> no memory issues at all.
>

Great :-)

Cheers,

*Cyril ADRIAN **(from office)
http://www.cadrian.net/cyril *
Share with me: [image: Twitter] <http://www.twitter.com/cadbart> [image:
LinkedIn] <http://fr.linkedin.com/in/cadrian> [image: Google

Calendar]<https://www.google.com/calendar/embed?src=1t93vvvrdc26ee0f83p0cunj60%40group.calendar.google.com&ctz=Europe/Paris>
[image:
Google Plus] <https://plus.google.com/100388810006463519079>
Contact me: [image: Google Talk] cyril.adrian@gmail.com
 [image: Twitter] <http://twitter.com/cadbart> Latest tweet: [Liberty]
http://t.co/xD9EgrtQ Cyril Adrian - more debug (commented out because it
takes a lot of CPU)
Follow @cadbart <http://twitter.com/cadbart> Reply

<http://twitter.com/?status=@cadbart%20&in_reply_to_status_id=127839909544337400&in_reply_to=cadbart>
Retweet

<http://twitter.com/?status=RT%20%40cadbart%3A%20%5BLiberty%5D%20http%3A%2F%2Ft.co%2FxD9EgrtQ%20Cyril%20Adrian%20-%20more%20debug%20(commented%20out%20because%20it%20takes%20a%20lot%20of%20CPU)>
 22:13 Oct-22<http://twitter.com/cadbart/statuses/127839909544337409>
  Get this email app!

<http://www.wisestamp.com/apps/twitter?utm_source=extension&utm_medium=email&utm_term=twitter&utm_campaign=apps>

 [image: Google Plus] <http://plus.google.com/100388810006463519079> My
latest G+: Dizzying but invisible depthYou just went to the Google home
page.... <http://plus.google.com/100388810006463519079/posts/RDiBNaX2XA7/>
My G+ <http://plus.google.com/100388810006463519079> -
Posts<http://plus.google.com/100388810006463519079/posts/>- Add
to Circles <http://plus.google.com/100388810006463519079/about/> - 09:32
Oct-17 <http://plus.google.com/100388810006463519079/about/>
  Get this email app!

<http://www.wisestamp.com/apps/plus?utm_source=extension&utm_medium=email&utm_term=plus&utm_campaign=apps>

[image: My QR VCard]
  Get a signature like this.

<http://r1.wisestamp.com/r/landing?promo=19&dest=http%3A%2F%2Fwww.wisestamp.com%2Femail-install%3Futm_source%3Dextension%26utm_medium%3Demail%26utm_campaign%3Dpromo_19>
CLICK

HERE.<http://r1.wisestamp.com/r/landing?promo=19&dest=http%3A%2F%2Fwww.wisestamp.com%2Femail-install%3Futm_source%3Dextension%26utm_medium%3Demail%26utm_campaign%3Dpromo_19>

Re: [libertyeiffel] Autorelative references and Liberty parser

From:
Paolo Redaelli
Date:
2011-10-24 @ 14:20
Il 24/10/2011 11:17, Cyril ADRIAN ha scritto:
> 2011/10/23 Paolo Redaelli <paolo.redaelli@gmail.com 
> <mailto:paolo.redaelli@gmail.com>>
>
>>     Why do you need more instances?
>     To run 6 times faster on an Core i7 and 12 times on dual Xeons
>     than current SE.... :)
>
>
> Parallelizing the core of compile_to_c is a big work.
> Don't start with the parser, I think it's not worth it (it is really 
> efficient and takes a really small portion of overall time).
>
> Better try to distribute tasks by adding synchronization points to 
> specialize_in, specialize_thru, specialize_2.
> Look at FEATURE_ACCUMULATOR, ANONYMOUS_FEATURE_MIXER. Those two 
> classes are the actual heart and liver of SmartEiffel.
>
> Another possibility is the C_PRETTY_PRINTER which generates the C 
> slices (happily the C generation now fully lives in 
> src/smarteiffel/generation/c).
Do we have any profiling of the compiler running on a noticeable code base?
I guess I shall start with C_PRETTY_PRINTER, it seems easier: I was 
looking for plain iterating over a collection, and I found it in 
SMART_EIFFEL.live_type_map which is iterated over in C_PRETTY_PRINTER's 
prepare_introspection and prepare_routine or the whole C_LIVE_TYPE_COMPILER.
I have to study SE code-base a little more, my wild guess was to write a 
variant of FAST_ARRAY[LIVE_TYPE] with parallelized iterating 
feautues...but it can't be that simple...

>     When dealing with multi-threading ømq approach fits very well with
>     Eiffel philosophy: threads never exchange data, they send each
>     other messages; just add the requirement that the sender owns the
>     message and that it will not release it until the receiver
>     answered it (as soon as possible) and you get no memory issues at all.
>
>
> Great :-)
My AmigaOS heritage taught me that a single address space is not evil as 
long as you communicate between threads using messages, in our case 
those provided by ømq. So let's see what make out runtime not thread 
safe. So far we know for sure that there are some static variables in 
the run time; so far I found that:

  * in deep_twin.c there is the hash table "se_deep_twin_memory" and the
    counter "se_deep_equal_start_counter". I think that using INTERNALs
    we may reimplement this in pure Eiffel
  * in base.c the runtime handler se_runtime_handler_t
  * the Garbage Collector is mostly non thread-safe; I haven't yet
    studied it, so I cannot say anything except that we may - at least
    temporary -use Boehm's conservative garbage collector (package
    libgc-dev in Ubuntu) which has been usable in SmartEiffel as far as
    I remember and it is thread safe. Not an optimal solution but
    feasible for a while
  * profile and sedb would require extensive changes

Am I missing something else?
I wrote more than a month ago that I would re-implementi deep_twin; I 
still don't have anything to show. I got busy boostrapping my teaching 
effort to architects at polimi and now I'm a little more free.

Re: [libertyeiffel] Autorelative references and Liberty parser

From:
Cyril Adrian
Date:
2011-10-24 @ 19:32
2011/10/24 Paolo Redaelli <paolo.redaelli@gmail.com>

> Do we have any profiling of the compiler running on a noticeable code base?
>
>

There were, of old. I don't know what became of them, but the first thing to
look at is Dominique's published papers.


>  I guess I shall start with C_PRETTY_PRINTER, it seems easier: I was
> looking for plain iterating over a collection, and I found it in
> SMART_EIFFEL.live_type_map which is iterated over in C_PRETTY_PRINTER's
> prepare_introspection and prepare_routine or the whole C_LIVE_TYPE_COMPILER.
> I have to study SE code-base a little more, my wild guess was to write a
> variant of FAST_ARRAY[LIVE_TYPE] with parallelized iterating feautues...but
> it can't be that simple...
>

It can be, for the back-end at least. Note that specialization is not that
simple because there are dependencies between live types.

>    My AmigaOS heritage taught me that a single address space is not evil
> as long as you communicate between threads using messages, in our case those
> provided by ømq. So let's see what make out runtime not thread safe. So far
> we know for sure that there are some static variables in the run time; so
> far I found that:
>
>    - in deep_twin.c there is the hash table "se_deep_twin_memory" and the
>    counter "se_deep_equal_start_counter". I think that using INTERNALs we may
>    reimplement this in pure Eiffel
>    - in base.c the runtime handler se_runtime_handler_t
>     - the Garbage Collector is mostly non thread-safe; I haven't yet
>    studied it, so I cannot say anything except that we may - at least temporary
>    -use Boehm's conservative garbage collector (package libgc-dev in Ubuntu)
>    which has been usable in SmartEiffel as far as I remember and it is thread
>    safe. Not an optimal solution but feasible for a while
>    - profile and sedb would require extensive changes
>
> Am I missing something else?
>

At least one I became aware of just recently: the exception contexts.

BTW I fixed exception handling with rescue clauses (it was quite flawed).

I wrote more than a month ago that I would re-implementi deep_twin; I still
> don't have anything to show. I got busy boostrapping my teaching effort to
> architects at polimi and now I'm a little more free.
>

Take your time, Rome was not built in a day ;-)

Meanwhile, I'm still coding the interpreter. Today was the contracts
handling session :-)
Note that I took some attention to parallel executions (one "just" has to
create more RUNNER_PROCESSOR instances although I guess RUNNER_MEMORY will
need some pampering for the dream to come true).

Cheers,

*Cyril ADRIAN*
*http://www.cadrian.net/cyril*
[image: Facebook] <https://www.facebook.com/cyril.adrian> [image:
Twitter]<https://twitter.com/cadbart> [image:
LinkedIn] <http://fr.linkedin.com/in/cadrian> [image: Google
Plus]<https://plus.google.com/100388810006463519079/posts> [image:
Google 
Calendar]<https://www.google.com/calendar/embed?src=1t93vvvrdc26ee0f83p0cunj60%40group.calendar.google.com&ctz=Europe/Paris>
Contact me: [image: Google Talk] cyril.adrian@gmail.com
 [image: Twitter] <http://twitter.com/cadbart> Latest tweet: [Liberty]
http://t.co/xD9EgrtQ Cyril Adrian - more debug (commented out because it
takes a lot of CPU)
Follow @cadbart <http://twitter.com/cadbart> Reply

<http://twitter.com/?status=@cadbart%20&in_reply_to_status_id=127839909544337400&in_reply_to=cadbart>
Retweet

<http://twitter.com/?status=RT%20%40cadbart%3A%20%5BLiberty%5D%20http%3A%2F%2Ft.co%2FxD9EgrtQ%20Cyril%20Adrian%20-%20more%20debug%20(commented%20out%20because%20it%20takes%20a%20lot%20of%20CPU)>
 22:13 Oct-22<http://twitter.com/cadbart/statuses/127839909544337409>
  Get this email app!

<http://www.wisestamp.com/apps/twitter?utm_source=extension&utm_medium=email&utm_term=twitter&utm_campaign=apps>

 [image: Google Plus] <http://plus.google.com/100388810006463519079> My
latest G+: Dizzying but invisible depthYou just went to the Google home
page.... <http://plus.google.com/100388810006463519079/posts/RDiBNaX2XA7/>
My G+ <http://plus.google.com/100388810006463519079> -
Posts<http://plus.google.com/100388810006463519079/posts/>- Add
to Circles <http://plus.google.com/100388810006463519079/about/> - 09:32
Oct-17 <http://plus.google.com/100388810006463519079/about/>
  Get this email app!

<http://www.wisestamp.com/apps/plus?utm_source=extension&utm_medium=email&utm_term=plus&utm_campaign=apps>

[image: My QR VCard]
  Get a signature like this.

<http://r1.wisestamp.com/r/landing?promo=17&dest=http%3A%2F%2Fwww.wisestamp.com%2Femail-install%3Futm_source%3Dextension%26utm_medium%3Demail%26utm_campaign%3Dpromo_17>
CLICK

HERE.<http://r1.wisestamp.com/r/landing?promo=17&dest=http%3A%2F%2Fwww.wisestamp.com%2Femail-install%3Futm_source%3Dextension%26utm_medium%3Demail%26utm_campaign%3Dpromo_17>