Re: [pyti] Short term objectives
- From:
- Feld Boris
- Date:
- 2011-05-25 @ 14:41
On mardi 24 mai 2011 at 17:51, Alexis Métaireau wrote:
Hello team,
>
Hi
>
>
> We need a kind of schedule to be sure that everything is on the rails
> and to give us objectives on the short term.
>
> Considering my last year Google Summer of Code, it goes too fast. You
> will probably run out of time without realising it. It is good to have
> you both on this project and I'm confident that we will have something
> working at the end of the summer but we need to be sure where we are
> heading and at what speed.
>
> To rephrase what we said before, we want at the end of the summer a
> master/slave continuous integration system. It should focus on python
> projects and be able to provide reports about different metrics. Those
> reports will be available through a central API.
>
> Doing things sequentially is probably an easy thing to plan but it also
> let all the adjustments to do for the end of the journey. I would like
> you to, in one month, already have a prototype working. Let's take one
> feature, say running the tests, and work on a simple implementation of it.
>
> This does not mean having a complete working project. It will allow us
> to see how you are working together; for you to foresee the different
> challenges you will have to go though and for all of us this put a short
> term deadline which is kinda exciting.
>
> Please, let me know what do you think about this.
>
> Boris, considering the report you made on the different tasks managers,
> I have a few questions about your pet project: when do you plan to work
> on the documentation? Is there some examples already out? and do you
> think the project is mature enough to be used as a tool in our case?
I planned to work on the documentation when it will be enough stable and
when enough features has been implemented (it should not remain a lot of
work).
Some examples are available in the unittests files (TestManager.py).
I think that the project is ready, maybe one or two features that i
discovered during making the report should be useful (like Resources
Control in narval and Fine Configuration On Failure in buildbot).
>
>
> Yeswanth, we don't have any new about your report so far (Or maybe I
> missed it?!). If you have difficulties to accomplish it, please let us
> now. It is not a problem and I can even try to free some of my time to
> help you on that. I remember myself last year and I remember being lost
> with all those new information. The most important thing is to let now
> the others about that. We are here to help :-)
>
> I will not be available most of the upcoming Thursday afternoon, what do
> you think about putting this meeting on the beginning of the week, say
> thuesday afternoon/evening ?
Agree, i'm available monday at 19:00 UTC, tuesday at 19:00 UTC and
wednesday at 19:00 UTC. If possible i prefer to avoid the tuesday evening.
>
>
> All the best,
> --
> Alexis
> Cheers,
FELD Boris
Re: [pyti] Short term objectives
- From:
- Alexis Métaireau
- Date:
- 2011-05-25 @ 19:24
On 05/25/2011 04:41 PM, FELD Boris wrote:
> I planned to work on the documentation when it will be enough stable and
when enough features has been implemented (it should not remain a lot of
work).
> Some examples are available in the unittests files (TestManager.py).
> I think that the project is ready, maybe one or two features that i
discovered during making the report should be useful (like Resources
Control in narval and Fine Configuration On Failure in buildbot).
Hmm, are those things really a part of the task management? Could you be
a bit more chatty about those two features?
You are talking about "communication between tasks". What are you
thinking of? About Narval, you're talking about POO. What is not POO in
the current code? How can it be enhanced?
Some of your bullet points are not well introduced and so I have a hard
time understanding what they are about. For instance:
> Dependencies: direct and output mode supported (4/4).
> Fine configuration of what doing if a step failed or emits warnings (3/3).
What are direct and output modes? What do you consider a "fine
configuration of tasks"?
Also, there is a paragraph in your blogpost which is directly written in
french ;-)
--
Alexis
Re: [pyti] Short term objectives
- From:
- Feld Boris
- Date:
- 2011-05-25 @ 20:24
On mercredi 25 mai 2011 at 21:24, Alexis Métaireau wrote:
On 05/25/2011 04:41 PM, FELD Boris wrote:
> > I planned to work on the documentation when it will be enough stable
and when enough features has been implemented (it should not remain a lot
of work).
> > Some examples are available in the unittests files (TestManager.py).
> > I think that the project is ready, maybe one or two features that i
discovered during making the report should be useful (like Resources
Control in narval and Fine Configuration On Failure in buildbot).
>
> Hmm, are those things really a part of the task management? Could you be
> a bit more chatty about those two features?
Resources Control are a way to stop a task if it reach a resource limit
(CPU time, memory quantity).
Fine Configuration is something i discover with BuildBot, indeed when you
configure a task you can configure how to catch failure and warnings :
http://buildbot.net/buildbot/docs/latest/Common-Parameters.html#Common-Parameters
>
>
> You are talking about "communication between tasks". What are you
> thinking of? About Narval, you're talking about POO. What is not POO in
> the current code? How can it be enhanced?
When i'm talking about "communication between tasks", it's simple output
of some tasks (raw data) can be used by other tasks as input.
In order to understand what i say that narval need POO, you can take a
look at sample tasks in this file :
http://hg.logilab.org/cubes/narval/file/dd50a034562f/plugins/basic.py
>
>
> Some of your bullet points are not well introduced and so I have a hard
> time understanding what they are about. For instance:
>
> > Dependencies: direct and output mode supported (4/4).
> > Fine configuration of what doing if a step failed or emits warnings (3/3).
>
> What are direct and output modes? What do you consider a "fine
> configuration of tasks"?
>
Direct mode : A dependency need that B executed before it. So task order
is : B -> A
Output mode (related to communication between tasks) : A dependency need D
input (or raw data), B produces D data during execution. Task order is the
same : B -> A, but it's more modular. For example, B and C can produce D
data, but B and C don't work on the same platform.
> Also, there is a paragraph in your blogpost which is directly written in
> french ;-)
Oops, nobody see it ^^
>
>
> --
> Alexis
> ---
Boris
Re: [pyti] Short term objectives
- From:
- yeswanth swami
- Date:
- 2011-05-25 @ 17:17
2011/5/24 Alexis Métaireau <alexis@notmyidea.org>
> Hello team,
>
>
Hi
> We need a kind of schedule to be sure that everything is on the rails
> and to give us objectives on the short term.
>
> Considering my last year Google Summer of Code, it goes too fast. You
> will probably run out of time without realising it. It is good to have
> you both on this project and I'm confident that we will have something
> working at the end of the summer but we need to be sure where we are
> heading and at what speed.
>
> To rephrase what we said before, we want at the end of the summer a
> master/slave continuous integration system. It should focus on python
> projects and be able to provide reports about different metrics. Those
> reports will be available through a central API.
>
> Doing things sequentially is probably an easy thing to plan but it also
> let all the adjustments to do for the end of the journey. I would like
> you to, in one month, already have a prototype working. Let's take one
> feature, say running the tests, and work on a simple implementation of it.
>
> This does not mean having a complete working project. It will allow us
> to see how you are working together; for you to foresee the different
> challenges you will have to go though and for all of us this put a short
> term deadline which is kinda exciting.
>
> Please, let me know what do you think about this.
>
I think it is a good idea to have a prototype in between. We would get a
good
idea on how things are heading.
>
> Boris, considering the report you made on the different tasks managers,
> I have a few questions about your pet project: when do you plan to work
> on the documentation? Is there some examples already out? and do you
> think the project is mature enough to be used as a tool in our case?
>
> Yeswanth, we don't have any new about your report so far (Or maybe I
> missed it?!). If you have difficulties to accomplish it, please let us
> now. It is not a problem and I can even try to free some of my time to
> help you on that. I remember myself last year and I remember being lost
> with all those new information. The most important thing is to let now
> the others about that. We are here to help :-)
>
I updated the wiki. ( Look after Boris' proposal ) . It would be great if I
could get
a feedback on that before the next meet. Would like a discussion with you
(other than the usual meet).
>
> I will not be available most of the upcoming Thursday afternoon, what do
> you think about putting this meeting on the beginning of the week, say
> thuesday afternoon/evening ?
>
> All the best,
> --
> Alexis
>
I am ok with any day. I am just working on Gsoc project this summer .
--
Cheers,
Yeswanth
Re: [pyti] Short term objectives
- From:
- Alexis Métaireau
- Date:
- 2011-05-25 @ 19:13
On 05/25/2011 07:17 PM, yeswanth swami wrote:
> I updated the wiki. ( Look after Boris' proposal ) . It would be great if I
> could get
> a feedback on that before the next meet. Would like a discussion with you
> (other than the usual meet).
Hey,
I've read your blogpost and the update you've made on the wiki about
Master/Slave architectures.
Good work about the different strengths and weaknesses of the different
tools, I think you now know in what overall direction you want to head
now. Please, keep in mind that we have sometime hard time to read a page
like a wiki (and especially the updates you're making to it), so please
post a link on the mailing list when you added some new contents.
When you are talking about "message passing", how do you intend to do
such thing with python? What are the benefits of such a thing? Does it
overcomplicate the task? etc.
If I take all the points you've put in "findings", here are some random
thoughts:
* +1 for a queuing system. We obviously need something that is able to
handle a lot of load and which can process that sequentially.
* A system like condor is IMO not needed here, we just need a simple
architecture to have multiple slaves controlled by a master.
* Did you find out why buildbot is using TCP connections? Is that an
error or is it justified?
Do you have some more things to share about the P2P protocols you're
talking about?
If you still want to talk to me, please also consider sending me emails.
That way we can talk asynchronously.
Cheers,
Alex
--
Alexis
Re: [pyti] Short term objectives
- From:
- yeswanth swami
- Date:
- 2011-05-26 @ 06:03
Hi,
2011/5/26 Alexis Métaireau <alexis@notmyidea.org>
> On 05/25/2011 07:17 PM, yeswanth swami wrote:
> > I updated the wiki. ( Look after Boris' proposal ) . It would be great if
> I
> > could get
> > a feedback on that before the next meet. Would like a discussion with you
> > (other than the usual meet).
> Hey,
>
> I've read your blogpost and the update you've made on the wiki about
> Master/Slave architectures.
>
> Good work about the different strengths and weaknesses of the different
> tools, I think you now know in what overall direction you want to head
> now. Please, keep in mind that we have sometime hard time to read a page
> like a wiki (and especially the updates you're making to it), so please
> post a link on the mailing list when you added some new contents.
>
> When you are talking about "message passing", how do you intend to do
> such thing with python? What are the benefits of such a thing? Does it
> overcomplicate the task? etc.
>
Actually on second thoughts, I have come to realize that message passing
is unnecessary. For communication request/response should be sufficient
through
which data can be sent.
>
> If I take all the points you've put in "findings", here are some random
> thoughts:
>
> * +1 for a queuing system. We obviously need something that is able to
> handle a lot of load and which can process that sequentially.
>
Any suggestion for implementation of queue ?
> * A system like condor is IMO not needed here, we just need a simple
> architecture to have multiple slaves controlled by a master.
> * Did you find out why buildbot is using TCP connections? Is that an
> error or is it justified?
>
> Having a TCP connection is a drawback for them. It leaves a connection open
for many
hours and to identify a connection error is very difficult. They are
planning for a new protocol
to overcome this drawback.
> Do you have some more things to share about the P2P protocols you're
> talking about?
>
>
I should mention here that Bitten's master/slave protocol is what I looking
at.
http://bitten.edgewall.org/wiki/MasterSlaveProtocol
(The diagram in the link is not very accurate w.r.t our project)
A modified version of this protocol is what I am proposing to be used in the
project.
I should write some details about it.
<http://bitten.edgewall.org/wiki/MasterSlaveProtocol>
> If you still want to talk to me, please also consider sending me emails.
> That way we can talk asynchronously.
>
> Cheers,
> Alex
>
>
> --
> Alexis
>
--
Cheers,
Yeswanth
Re: [pyti] Short term objectives
- From:
- Alexis Métaireau
- Date:
- 2011-05-26 @ 11:42
On 05/26/2011 08:03 AM, yeswanth swami wrote:
> Actually on second thoughts, I have come to realize that message passing
> is unnecessary. For communication request/response should be sufficient
> through
> which data can be sent.
That's what was my feelings at the first glance, happy you've found this
as well in the end.
> Any suggestion for implementation of queue ?
There are a lot of queues implementations available on PyPI or
otherwise; i haven't tried them out so far so it's a bit hard to say
which one to use. What we can do is to define our needs.
What we need is a simple thing: sending messages to different slaves
from the master, and be sure that they are treated sequentially.
Ideally, all the slaves receives the message on only one finally proceed
it. Slaves may be able to handle more than only one process at a time.
The standard python library defines a queueing library [0], which may be
useful for us but which is not taking in consideration the communication
problem between different servers (it's intended to use on only one
python program).
On the other side, AMQP defines a protocol which can be useful for
queues [1], with some implementations already out there: RabbitMQ,
ZeroMQ and others.
There also are some implementations relying on third party softwares
such as redis [2] or some other ones based on twisted [3]. The advantage
of the latter one is that it is pure python based, even if I tend to
think that twisted is somehow too complicated for what we try to
achieve. Having a queuing system on top of redis seems something which
can solve our problems easily but on the other hand means that we will
be depending on redis.
Do you have any thoughts on that? Have you also done some search?
[0] http://docs.python.org/library/queue.html
[1] http://www.zeromq.org/whitepapers:amqp-analysis
[2] http://pypi.python.org/pypi/hotqueue/0.2.5
http://pypi.python.org/pypi/redis_queue/0.5
[3] http://pypi.python.org/pypi/AsynQueue/0.3
> Having a TCP connection is a drawback for them. It leaves a connection open
> for many
> hours and to identify a connection error is very difficult. They are
> planning for a new protocol
> to overcome this drawback.
This does not explain why their were using it at the beginning. Alright,
this is a problem for them, but why did they made this choice at the
beginning? Maybe was it solving particular problems for them?
I guess it can be useful to have real time feedback about what's going
on on the server for really long processes.
> A modified version of this protocol is what I am proposing to be used in the
> project.
> I should write some details about it.
Yep please go ahead :)
--
Alexis