dltk project

jgangemi wrote on Wed Nov 22 23:29:25 CET 2006:

have you seen this? http://www.eclipse.org/proposals/dltk/

the topic of the dbgp open protocol came up at work today while discussing
the need/desire to be able to debug perl code remotely, and i came across
the dltk project once it was mentioned that there was an eclipse plugin
supporting dbgp.

overall, it looks as if the dltk project could offer us the infrastructure
to add a lot more functionality and features to epic, but at the moment,
they are only up to the m3 release, with a full 1.0 release scheduled for
the end of january 07.

i'm not sure if i'll have the time this weekend, but i'm going to do a bit
more investigative work on this. 

ppl at my company are already thinking about writing the perl stub for the
protocol (it currently exists for active state, but not "unix" perl), and
it would definitely be nice if epic could integrate w/ it in some way (there
was also talk of writing a wrapper around the stub so it behaves just like
the current perl debugger interface)

jploski wrote on Thu Nov 23 00:08:35 CET 2006:

I had seen the DLTK project before, but did not investigate it closely.

Now I also had a quick look at the DBGP protocol. As the only available
implementation seems to be for PHP, before committing any resources to the
protocol's implementation, it would be interesting to do a simple performance
test of this implementation - how long does it take to retrieve and display
multiple variables with large deeply nested data structures. By observing
the EPIC/Perl protocol (for example by turning on the "debugger console"
preference), you will see that quite huge amounts of textual data are retrieved
after each step currently. A good prospective protocol should not transfer
data in a way which takes substantially longer to process than the current
one (which is already too slow). I am skeptical about the XML format  used
by DBGP. JDWP (the protocol used by the JVM) might be a binary protocol
for a good reason.

jgangemi wrote on Thu Nov 23 17:14:34 CET 2006:

i'm definitely not proposing spending any time developing a perl implementation
of the dbgp protocol - however ppl that i work with are thinking about it.
we encrypt our perl modules and load them into memory so that they never
reside on disk as a way to protect our IP, however it makes debugging extremely
difficult for due to a number of reasons (code is on remote host, it's not
on disk, etc). 

they had discussed writing the protocol implementation (apparently komodo
and other editors already support it) and potentially also writing a wrapper
around it so it behaves just like the perl debugger. if they do go ahead
with this, regardless of speed, it will be something i'll want epic to support
- and could be accomplished by providing some kind of debugger extension
point.

right now my interest is in what dltk will provide, which seems to be a
lot - i've really been wanting a more jdt like interface for the build path
(it's already an rfe) and a better package explorer view then just the flat
folder system - but those are no small undertaking, but dltk already has
support for them. 

of course, dltk requires eclipse 3.2, but that issue can be addressed once
dltk's offerings are flushed out.

jploski wrote on Thu Nov 23 18:49:16 CET 2006:

Do I understand you correctly that the debugged Perl source code is available
on the developer's machine, but not on the machine where it executes? That
would mean that a local version of a source file can still be opened on
the EPIC side when the remote debugger hits a breakpoint?

If so, I think that the proper way to address this feature would be to start
with a plain vanilla Perl debugger on a remote machine (listening on a TCP
port there) and make EPIC work with it. I have never tried out the remote
debugging feature of EPIC, but I think it follows this general idea. If
that works, you could swap in your wrapper implementation in place of the
real debugger - so far I see no need to change EPIC in a way specific to
your environment, as EPIC does not care about the implementation behind
the TCP socket.

However, maybe you would like to implement another launch configuration
type (like the current "Perl Remote" one), which is indeed not generally
useful. In this case, I think that we could specify the interface between
your plug-in and org.epic.debug simply through a list of supported launch
configuration attributes. If more elaborate extension mechanisms are needed
in org.epic.debug, that's no problem either, it would be nice if they were
abstract enough to be generally useful.

So far EPIC does not define any extension points. It would be very much
in the spirit of Eclipse to define ones and encourage others to build upon
them. We would have to be careful about design, though, because as soon
as we export extension points, they become published interfaces that we
need to care about "forever" - a whole new dimension of backwards compatibility
issues.

BTW, I had an opportunity to try out the CGI debugger under Windows today
and it was a big disappointment (hanging Eclipse, ignoring breakpoints,
leaving around perl.exe processes taking 100% CPU time).

jgangemi wrote on Thu Nov 23 19:14:09 CET 2006:

the code is loaded in memory, so the debugger would need to send back line
number information to the client, who has a copy of the source locally -
much like debugging a remote tomcat session. i do agree that the socket
impl would be the easiest thing to try, but they (co-workers) didn't really
want to listen to that and were insistent upon just going straight to a
dbpg implementation. 

i haven't had a real need for the debugger as of yet, print stmts have been
working just fine when i've encountered a problem, but perhaps i should
give another look into epic's remote debugging support. i'm sure i'm going
to wish i had a way to remotely debug code. if only there were more hours
in the day...

jgangemi wrote on Fri Dec 15 18:24:58 CET 2006:

i thought i'd give an update on my dltk findings, while i am enjoying the
challenge of figuring out how everything works and fits together, i don't
want to get trapped too far in a rabbit hole...

the toolkit itself is pretty self explanitory - aside from providing a uniform
framework for scripted languages, it also provides a lot of the base functionality
that is needed to add more "advanced" features
(ie: jdt style buildpaths) that have been asked for by others (and some
that i would like to see myself)

at this point, i've managed to put together a very basis prototype to get
a feel for what is going on and how things work - currently creating a new
project, managing build paths (this part rocks, i love the flat view i get
for all my perl packages), basic coloring/partitioning, etc.
  
- in all of the available language implementations (except ruby, which is
now using the jruby), the parser
is used to build an AST model. in order to provide some of the more "advanced"
functionality (such as call heirarchy, etc) , it seems that the parser would
need to be completely reworked, or another parser created to do the AST
model work. the AST model is also used to provide the outline view, which
the current parser can still handle. 

- partitioning/coloring are handled by using a rule based scanner implementation,
the same as the jdt. the scanner impl allows partitioning and coloring to
happen independently, which i would consider to be a good thing b/c it offers
more flexibility. decoupling these two things might also make it easier
to build in more complex folding (i'd like to see if, while, etc stmts fold).
it seems that it should be possible to still
rely on the lexer to provide this information instead of the rules, but
that would mean the document
is parsed twice by the lexer, once for "display", the other for AST generation.
given that the document is parsed twice regardless, that might not really
make a difference.

- the current multilexer does work w/ the dltk implementation -instead of
relying on the document, it
just gets the contents instead. this does "break" some of the curly brace,
operator, etc positioning logic that was occurring - but that might not
be needed w/ the rule based scanners.

overall - i'm pretty impressed with what is provided by the toolkit and
what i've been able to put together with relatively minimal efforts. the
project has been accepted by the eclipse foundation, so it has serious backing,
i think we should give some serious consideration to taking advantage of
the toolkit. (on a side note, i also created a new editor icon that i hope
we can switch too regardless of the direction that is taken)

if you're interested, i can put everything i've got so far into some kind
of package that you could check out. everything is in a brand new namespace,
so it won't conflict w/ anything (and can easily be renamed, etc) - although
installing the tcl, python, and/or ruby plugins would give a grander scope.

jgangemi wrote on Fri Dec 15 18:39:21 CET 2006:

you can grab a copy of the prototype here: http://www.scriptkitty.org/perlipse.zip

you'll also need to grab the lastest dltk sources from here: :pserver:anonymous@eclipsedltk.org:/opt/dltk

everything compiles against cvs as of last night, but if you encounter any
compile errors, let me know.

jgangemi wrote on Wed Dec 20 19:57:03 CET 2006:

not sure if you gave this a glance yet, but i've updated the prototype w/
a run delegate implementation, so scripts can be executed. 

it seems that Active State's dbgp protocol is open source (perl artistic
license), and it looks like the folks at work will be incorporating that
into our embedded perl interpreter for remote debugging in the coming days.
given that i don't want to use komodo, i'm gonna need to move forward on
something that will support this sooner then later.

jploski wrote on Wed Dec 20 20:59:58 CET 2006:

I tried it today for the first time. It compiles fine, but an attempt to
create a "Perl Script" launch configuration leads to

java.lang.NullPointerException
at com.xored.dltk.debug.ui.launchConfigurations.MainLaunchConfigurationTab.initializeFrom(MainLaunchConfigurationTab.java:209)
at org.eclipse.debug.ui.AbstractLaunchConfigurationTabGroup.initializeFrom(AbstractLaunchConfigurationTabGroup.java:86)
at org.eclipse.debug.internal.ui.launchConfigurations.LaunchConfigurationTabGroupViewer.displayInstanceTabs(LaunchConfigurationTabGroupViewer.java:655)
at org.eclipse.debug.internal.ui.launchConfigurations.LaunchConfigurationTabGroupViewer$6.run(LaunchConfigurationTabGroupViewer.java:548)
at org.eclipse.swt.custom.BusyIndicator.showWhile(BusyIndicator.java:67)
at org.eclipse.debug.internal.ui.launchConfigurations.LaunchConfigurationTabGroupViewer.inputChanged(LaunchConfigurationTabGroupViewer.java:565)
at org.eclipse.debug.internal.ui.launchConfigurations.LaunchConfigurationTabGroupViewer.setInput(LaunchConfigurationTabGroupViewer.java:529)
at org.eclipse.debug.internal.ui.launchConfigurations.LaunchConfigurationsDialog.handleLaunchConfigurationSelectionChanged(LaunchConfigurationsDialog.java:928)
at org.eclipse.debug.internal.ui.launchConfigurations.LaunchConfigurationsDialog$4.selectionChanged(LaunchConfigurationsDialog.java:547)
at org.eclipse.jface.viewers.StructuredViewer$3.run(StructuredViewer.java:839)
at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:37)
at org.eclipse.core.runtime.Platform.run(Platform.java:843)
at org.eclipse.ui.internal.JFaceUtil$1.run(JFaceUtil.java:44)
at org.eclipse.jface.util.SafeRunnable.run(SafeRunnable.java:149)
at org.eclipse.jface.viewers.StructuredViewer.firePostSelectionChanged(StructuredViewer.java:837)
at org.eclipse.jface.viewers.StructuredViewer.handlePostSelect(StructuredViewer.java:1143)
at org.eclipse.jface.viewers.StructuredViewer$5.widgetSelected(StructuredViewer.java:1163)
at org.eclipse.jface.util.OpenStrategy.firePostSelectionEvent(OpenStrategy.java:236)
at org.eclipse.jface.util.OpenStrategy.access$4(OpenStrategy.java:230)
at org.eclipse.jface.util.OpenStrategy$3.run(OpenStrategy.java:404)
at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:35)
at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:123)
at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:3143)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:2845)
at org.eclipse.jface.window.Window.runEventLoop(Window.java:820)
at org.eclipse.jface.window.Window.open(Window.java:796)
at org.eclipse.debug.internal.ui.launchConfigurations.LaunchConfigurationsDialog.open(LaunchConfigurationsDialog.java:1086)
at org.eclipse.debug.ui.DebugUITools$1.run(DebugUITools.java:383)
at org.eclipse.swt.custom.BusyIndicator.showWhile(BusyIndicator.java:67)
at org.eclipse.debug.ui.DebugUITools.openLaunchConfigurationDialogOnGroup(DebugUITools.java:387)
at org.eclipse.debug.ui.DebugUITools.openLaunchConfigurationDialogOnGroup(DebugUITools.java:329)
at org.eclipse.debug.ui.actions.OpenLaunchDialogAction.run(OpenLaunchDialogAction.java:80)
at org.eclipse.debug.ui.actions.OpenLaunchDialogAction.run(OpenLaunchDialogAction.java:99)
at org.eclipse.ui.internal.PluginAction.runWithEvent(PluginAction.java:254)
at org.eclipse.ui.internal.WWinPluginAction.runWithEvent(WWinPluginAction.java:229)
at org.eclipse.jface.action.ActionContributionItem.handleWidgetSelection(ActionContributionItem.java:539)
at org.eclipse.jface.action.ActionContributionItem.access$2(ActionContributionItem.java:488)
at org.eclipse.jface.action.ActionContributionItem$5.handleEvent(ActionContributionItem.java:400)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:66)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1085)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:3166)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:2842)
at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.java:1914)
at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:1878)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:419)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:149)
at org.eclipse.ui.internal.ide.IDEApplication.run(IDEApplication.java:95)
at org.eclipse.core.internal.runtime.PlatformActivator$1.run(PlatformActivator.java:78)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:92)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:68)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:400)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.eclipse.core.launcher.Main.invokeFramework(Main.java:336)
at org.eclipse.core.launcher.Main.basicRun(Main.java:280)
at org.eclipse.core.launcher.Main.run(Main.java:977)
at org.eclipse.core.launcher.Main.main(Main.java:952)

jgangemi wrote on Wed Dec 20 22:05:36 CET 2006:

odd, i can't reproduce that. i just synced against cvs, and everything works
fine for me. i just re-zipped everything in case i missed something the
first time around. it's possible i forgot to do something, but i can't cause
that to error to occur even w/ a brand new workspace.

perhaps a stupid question, but you are giving this a whirl in 3.2, right?
you've also created a project and a sample test file to execute? i can't
see why it would be breaking since the method i need to implement to validate
the project is called after the EditorInput is obtained. 

did you see the new editor icon? i'm quite proud of that (although i had
some help making it)

jploski wrote on Wed Dec 20 22:23:52 CET 2006:

Yes, I'm trying with 3.2. More details: the exception only occurs when I
do the following:
1. Open a Perl file in an EPIC (not DLTK!) project
2. Go to Run... and try creating a new launch configuration

It does not occur with a Perlipse Project. I managed to run a sample script
without problems (no setting of breakpoints yet, right?)

The icon looks clean and nice, but the 'P' could be misinterpreted for PHP
or Python, while a camel is a camel.

jgangemi wrote on Wed Dec 20 22:43:06 CET 2006:

ahh - that makes sense then, although i'm not sure why it didn't handle
the scenario gracefully. no setting of breakpoints yet - that was a bear
just to get into epic and i didn't really spend a lot of time on 'fluff',
i just wanted to get a bunch of the basics put together as a proof of concept.

damn python and php for starting w/ a 'p' - i wonder how hard it would be
to stick the camel inside the editor icon frame...

jploski wrote on Wed Dec 20 22:47:47 CET 2006:

Just make sure PETA do not find out.

jploski wrote on Wed Dec 20 21:58:07 CET 2006:

I looked a bit more into the DLTK project (web site, source code), as a
potential migration target for EPIC. AFAICS, there are currently no documented
published APIs and no documented extension points. Hopefully this is just
a matter of work in progress and not a lack of foresight. Providing reference
implementations is fine, but I'd expect specifications on par with those
produced by the Platform/JDT projects. After all, DLTK's ambitions seem
to lie somewhere between the JDT's and the Platform's.

jgangemi wrote on Wed Dec 20 22:19:51 CET 2006:

yeah - i mentioned that to them and the documentation is on the road map,
but not until RC stage of things - mostly i've been just digging around
and looking at the reference implementation to figure things out.

the project has been approved/accepted by eclipse folks, so i'm pretty sure
it's just an oversight at the moment. i can't imagine the eclipse folks
would let something under their umbrella release if it wasn't up to snuf.

jploski wrote on Wed Dec 20 22:37:03 CET 2006:

> i can't imagine the eclipse folks would let something under their umbrella
release if it wasn't up to snuf.

Given the great count of Eclipse projects, I wouldn't be surprised at all
by varied quality and project governance. I don't want to name examples,
though...

Anyway, I intend to keep an eye on DLTK in 2007. :-)

jgangemi wrote on Wed Dec 20 22:48:46 CET 2006:

heh - good point.

well - the road map says the core (w/ tcl tools) will be release on jan
23rd, w/ the final coming by end of Q1/2007. based upon the needs of my
day job, i'm gonna keep up w/ the integration work, esp in the dbgp area
- and hopefully we'll be able to sync up at some point - otherwise we'll
have to figure out how to get epic to support that functionality as well.