uPortal Framework Developers Meeting

Arizona State University, Tempe, Arizona February 9-10, 2004

These notes were taken by Michael Oltz and put in his own words. They do not exhaustively represent every word that was said. Their intent is to show the topics discussed and the decisions made.

people present:

• Trent[on D.] Adams -- Athabasca U -- trenta@athabascau.ca
• Nancy Lee -- ASU -- nancy.lee@asu.edu
• Bob Miller -- ASU -- bob.miller@asu.edu
• Ron Page -- ASU -- ron.page@asu.edu
• Al Wold -- ASU -- alwold@asu.edu
• Dan Ellentuck -- Columbia U -- de3@columbia.edu
• Alex Vigdor -- Columbia U -- av317@columbia.edu
• Steve Barrett -- Cornell U -- smb1@cornell.edu
• Michael Oltz -- Cornell U -- mdo1@cornell.edu
• Michael Ivanov -- instructional media + magic (IMM) -- mvi@immagic.com
• Nate Johnson -- Indiana U -- natjohns@indiana.edu
• Jim Farmer -- JA-SIG and IMM. -- jxf@immagic.com
• John Bush -- r•smart -- jbush@rsmart.com
• Chris Coppola -- r•smart -- chris.coppola@rsmart.com
• Mike DeSimone -- r•smart -- mike.desimone@rsmart.com
• Bill Thompson -- Rutgers U -- wgthom@rutgers.edu
• Dmitriy Kopylenko -- Rutgers U -- dkopylenko@acs.rutgers.edu
• Mark Boyd -- SCT -- mboyd@sct.com
• Jan Nielsen -- SCT -- jnielsen@sct.com
• Mike Zackrison -- SCT -- mzackris@sct.com
• Dave Wallace -- U Delaware -- dwallace@udel.edu
• John Blakley -- President of Unicon, sat in on the Tuesday morning session.
• Nick Bolton -- Unicon -- nbolton@unicon.net
• Kevin Gary -- Unicon -- garyk@unicon.net
• Shawn Lonas -- Unicon -- shawn@unicon.net
• Ken Weiner -- Unicon -- kweiner@unicon.net
• Andrew Wills -- Unicon -- awills@unicon.net
• Susan Bramhall -- Yale U -- susan.bramhall@yale.edu
• Howard Gilbert -- Yale U -- howard.gilbert@yale.edu

MONDAY

uPortal and Sakai -- Jim Farmer

At last fall's developers meeting (at Cornell University),
1) It was agreed that uPortal should support JSR-168 (portlets) and WSRP (web services for remote portlets) soon
2) Chuck Severance from UMich mentioned that there should be some collaboration between uPortal developers and other efforts at Michigan, Indiana, and MIT.

At MIT, Jeff Merriman was leading the development of OSIDs (OKI Services Interface Definitions) which were defined as Java APIs. In September two people from the OSID project met at UMich with people from several universities and JA-Sig; out of that came a proposal. It was funded in December by the Mellon Foundation: Indiana U, UMich, Stanford, MIT -- a project to make course delivery and management and collaboration tools.

Originally UMich wanted somebody to put Pluto into Jetspeed. But they decided that uPortal may be a better platform to do this with respect to higher ed, because it has:
1) internationalization
2) Aggregated layouts
3) Groups and permissions -- it can be overridden to work differently

The second half of Sakai: tried to create a licensing arrangement to permit corporations to make extensions to it.

Should the OSID services be available only THROUGH the portal or also OUTSIDE of it?

The two projects compared:

"legacy" means, were there substantial existing projects that needed to be incorporated into the new project when it began?

The Cornell meeting was the first one where corporate folk outnumbered people from higher education.

Biggest implication for JA-SIG is we need to have JSR-168 by Sakai's schedule, "really fast" rather than at a more moderate pace.

Aggregated Layouts -- Michael Ivanov

Layout content management is related to roles and groups, and is implemented chiefly in the Java classes AggregatedLayoutManager and AggregatedLayoutStore.

There are many early limitations. Eventually there will be more ability to publish and subscribe, and editors. So far you have to create fragments in XML and run an ant target 'pushfragment' to put it in the db. So far fragments are all immutable, and they can only be tabs; you can't have a column or smaller fragment. The early limitations are in the manager and store.

Should we do the enhancement in uPortal 2.3 rather than in 3.0?

Pushed fragments -- can be added by administrators; who gets it is based on groups and permissions. 'Marking nodes' are put in the layout to indicate whether and where a fragment can be used. And there are different kinds of restrictions. Hidden, removable, immutable. And the 'priority range' -- what order does the item appear in the tab/column tree. Each manager of fragments gets a range of priorities that they can assign to the fragments they create. How to do this will be explained in the upcoming uPortal documentation.

In 3.0 he wants to introduce user modifiable fragments. There are complicated questions as to how the two kinds of fragments will interact.

There are also pulled fragments, that the user could be able to choose at will, but there is as yet no UI to get to them; the store and manager know how to handle them though.

Most of the default layouts in 2.2 consist of fragments only. This has caused some confusion; portal implementers expect them to be editable in the UI -- not yet!

There is a new target in uPortal called 'initportal' that runs all the initializations.

The chapters of the docs are checked in to CVS, not complete as yet.

There is a straightforward way to switch between AggregatedLayout and Simple. You change the properties for the layoutmanager and the layoutstore, back to the SimpleLayout versions, and you'd have to set everybody to the older theme.

Rutgers: is it worthwhile to migrate to 2.2 if you don't want Aggregated Layouts yet? You can't use the new theme with the old layouts. [the only thing is the marking nodes information is not in the old XML, so the users would not be able to edit anything] Peter wanted to put these nodes into the old layout but it's not finished yet.

Using the old themes with the new layouts. It would render, but nothing would read the restrictions. In the old layouts, the knowledge of where stuff can be added or moved is in the XSLT. In the new layouts, the knowledge is in the manager, and the layout asks the manager where it can put stuff.

Who wants either of these? Susan: it would be nice to have a conversion path. Steve: the UI is changing a lot and needs lots of staff involved. Jim F.: Jon Allen at IMM is writing user and admin docs. For sites who are just coming online they can go directly to 2.2. Peter thought the ability to create and edit fragments would rarely be used, by admins. Jim: It will be used a lot by faculty.

Susan: simplest conversion would be to go to aggregated layouts without integrated modes. Then change to the new UI later.

Ken: we need someone to work on an upgrade path.

Michael I: Could we get editors into 2.3 or 3.0?

Dan E: At Columbia only a small percent of users bother to modify their layouts.

Susan: Some sites have made customized themes not just skins. Will be hard to migrate.

Ken: A committee to work on migration?

Dave Wallace: What is the timeframe on getting a UI to edit?

Jim: Let's do it immediately. Ken: Doesn't want to commit due to where would we get the resources.

Alex: Is there a way to do it like  CuCMS does where you could use a channel to edit the XML and write it out from there?

Jim: There aren't many resources available to do this. Hazard of promising to do this then not doing it.

Bill T: Rutgers' emphasis right now is content, we don't want to spend time migrating now.

[break]

More about Sakai and 2.3/3.0 -- Jim Farmer

Jim Farmer then showed the slides of his presentation on the status of uPortal, as they finally had the projector working. These slides will be/are available online.

He spoke about remaining limitations of the new features. These are listed on the slides.

In uPortal 2.3 the initial implementation JSR-168 will be done without regard to performance. The preliminary WSRP (Web Services Remote Portlet) code from 2.2 will be completed.

In 3.0. Performance of JSR-168 will be improved. IChannel will still be supported and existing channels will still work.

Future uPortal priorities are likely to be:

• Stability
• Performance
• Features
• ----
• Product support
• Documentation
• Training

Future JA-SIG priorities are likely to be:

• Applications implemented as portlets or portal channels
• OSPI ePortfolio
• Sakai collaboration tools
• Services oriented architecture

JA-Sig priorities and activities represent interests, needs, and resources of the participants. There are changing needs in higher and further education, so the priorities for the uPortal group are changing.

He also showed slides by Lois Brook of Stanford University describing Sakai Project to U of California CIO's.

Sakai products:

Tool Portability Profile: standard for writing tools to extend the core apps.
A Technical description
Describe a common path forward.
A complete CMS with assessment tools
Enterprise services based portal
Workflow engine
Research support collaboration system.

Skin in the game
$4m total from the four institutions
$2.4m from Mellon Foundation

Jim quoted Brad Wheeler as saying "There is no Plan B." It goes as planned, very fast, or it fails.

Gartner: "By 2007, 80 percent of e-learning platform functionality will be available through open source (0.7 probability)"

The Sakai Educational Partners Program (SEPP)

Who can be in it:
Ed institutions only
Administrators for planning purposes
Adopters who need support
Developers who want to contribute

What they get:
Can participate in community
Tech support for partners
Knowledgebase
Tech docs and specifications.
Early access
Access to SEPP staff
Access to knowledgebase

Time base. Jim showed a slide giving dates, all in 2004.

First developers workshop for Sakai will be colocated with summer JA-SIG meeting, following it. [After the Tempe meeting it was found out that the date and location of that workshop may be changed] It will cover the "Tool Portability Profile" and what has been done to existing products to make them compliant.

According to a participant, the Federal govt has an educational authentication standards project going. This will recommend a federation rather than a single national authentication service. It will use "commercially available software" -- that suggests the Liberty Alliance. Federal govt has agreement with banking community to use their ATM-card based identification systems for identification.

Collaboration is not everyone doing the same thing, but rather, doing different things that work together.

[OSID is on Sourceforge under 'OKI']

There are two newsfeeds on the IMM site under uPortal. Watch them often.

Integrated Modes -- Ken Weiner

[already been mentioned but we can talk about it more.]

Since the projector is now working, Ken demonstrated integrated modes. The default pushfragments file is in properties/al. The pushfragment files refer to channels by their functional names. There is a new icon at the top 'sitemap' that shows a summary of your layout as a text outline. Ken wrote a very simple channel to choose the language in which to show the portal. There are currently three levels of language settings -- browser, session, and user. Not yet at channel level.

Steve B asked to demonstrate how to move a channel.

Susan: Top of the portal takes up too much of the screen. Ken: You could change the theme. with a new integratedmodes.xsl [may also have to change the skin] Ken: "This happens because Justin works on a very large monitor, so on his screen it looks really good."

From the theme xsl, there are two links to CSS, one is the uPortal skin for the page, and another that points to a 'portlet' CSS so there will be a skin to comply to JSR-168 which has its own CSS spec. [well, WSIA standard] We'll need a new CSS for each skin. For 2.3 and 3.0 we (JA-SIG) will be converting most of our channels to portlets, and a graphic designer will have to change the CSS used by the channel to be the portlet one.

Internationalization (i18n) -- Ken Weiner

Java locale preferences are specified at three levels. There is an algorithm which determines how the three levels override to result in one list. The browser strings are not easily parsable into Java locales. Nobody at the meeting indicated good familiarity with the issue. This needs addressing. There is an RFC: 2616 section 14.4 that specifies how the browser language settings are supposed to look.

The channel gets an ordered list of locales, and it's up to the channel to decide how to deliver content based on those locales.

There is a utility put into the XSLT util for internationalization that can recognize locales. [getTransformer method] This is what can select different XSLT files based on language.

Right now we have like, six different stylesheets if that one has been internationalized. And it will get worse as we add more languages. The building of the language variants should happen at build time. Can we get a better arrangement of this by 2.3? Right now even if a language specialist wants to keep their language current, there is no mechanism to do it.

Alex: I have a utility that inserts resource bundles into XML, which I wrote for CuCMS, if anyone wants to use that. Alex explained more about how his code works and pointed out the advantage to doing it that way.

Jim: threw in a caveat that a single English phrase can be translated various ways depending on the context.

The group began to feel that Alex's way to handle it may be what we want.

Jim: The original mentor of our i18n project is VP for i18n at Oracle. We should ask him what he thinks about that.

Jan: We may want to write our own resource bundle code to customize the behavior.

Shawn: Maybe the language-string reading could happen at runtime, calling out from XSLT to get them.

Alex: This is sort of what my code does already.

[lunch]

WSRP -- Ken Weiner

[he has a long bunch of slides about the topic that he wrote for doing uPortal 2.2 training; he only showed some of them in this session]

WSRP in uPortal does everything that Remote Channels did in 2.1. [Nobody present had tried it yet] It only works for some channels and not others. There are some bugs that need fixing. There are some known limitations mostly having to do with authentication and user identity.

Publish parameters -- the Service URL base. Don't forget the context name if needed. And the portlet handle name which in uPortal is the functional name. The registration interface in WSRP is optional and in 2.2 ken didn't implement it; regstration-required services won't work. The other optional interface is portal entity management. that isn't in here. One institution in UK was interested.

Next steps for WSRP in uPortal:
WSRP4J
Compatibility testing
Authentication
Consumer URL rewriting
File upload/downloading
Use of Registration Interface
Use of Portlet management interface
Portlet Browsing and Subscription

[all of these things need help, Ken will not be able to do them all]

Jim asked what is Cornell's interest in WSRP: Steve replied, he thinks people will eventually stop writing servlets as such and will write portlets instead, and we will need to prepare for that.

Jim: For people who are writing portlets, which will you use WSRP or JSR-168?

Bill T: WSRP = remote portlets, which in our experience means 'slow'. They more want local things.
Steve: You could still have a local WSRP.
Alex: If you write JSR-168 then you have WSRP.

Jim: In UK they write WSRP but in USA people seem to want JSR-168.
Ken: Well if they want to use Java then JSR-168 is the way to write a portlet.

Jim: a project will begin in UK on Feb 16 to implement a library system as a web service.

Ken really wants others to test it a lot to help shake out the bugs.

[Ken had given a basic training for uPortal 2.2, in France the previous week]

The people in the France training didn't like the way the portal controls who can get at a particular portlet. Right now if the portlet is permitted to the group Everyone, anyone can be a client of it. They wanted a way to limit this.

There was a discussion about what this meant and how we might offer what they want. What if uPortal requires registration in the future? Registration is a temporary thing similar to establishing a session; it is not a persistent thing. When you connect to a WSRP producer, you can find out whether the producer requires registration.

Jim: A university should decide on a consistent way to require registration, so that there's not a different way for every service.

More talk about portlets -- Apache Pluto -- Ken Weiner

In 2.3, the classic channels will be there, and there will be one more channel type that would be a portlet. Those channel types would go through an adapter so the Pluto API will look like an old channel.

In 3.0, Pluto would be 'it'; most of the provided channels will be JSR-168, and there will be an adapter to make the classic channel API look like a portlet.

The Pluto build makes three jars.
portlet API jar file - javax.portlet type of code, almost all interfaces. Likely to be stable
pluto1.0.jar -- the Pluto container -- meant to be imbedded in other people's portals.
portalImpl.jar -- the part that we're replacing -- this was a minimal thing written by the Pluto team for testing and demonstration.

Their sources are in several pieces
javax.portlet -- the portal API part -- not yet distributed by Sun.
org.apache.pluto -- the particular Pluto run code
org.apache.pluto.portalimpl -- this was where the portalImpl came from.
Their 'portlet entity' is like uPortal's 'subscribeID' -- an instance of the portlet for a particular client.

[Ken spent some time going over the arrangement of the Pluto source and the purpose of some of the packages and classes]

In 2.3, there will be the two jars mentioned above, which will go into the shared library directory, according to what servlet container you are using.

Seems to work on Tomcat and Resin but not on WebSphere.

There is a new package org.jasig.portal.container

In the StaticInformationProvider class there is some hard-coded information that needs more flexibility as to where it comes from.

Howard suggested a different way to implement this, having the Pluto context be separate from the other portal context.

Alex: That means you can use the Tomcat management facility to take a portlet up and down separately when upgrading them.

The Pluto spec says that you have to define a portlet in portlet.xml which is to be in WEB-INF. Each of the portlets will have names. The uPortal adapter will look up the portlet in this file by the provided I.D. [see ken's explanation in previous session of configuring such a channel]

Next:
-- change StaticInformationProvider to come through webapps to get information
-- implement DynamicInformationProvider so you can respond to controls in the portlet.

Ken thinks that much will be done in weeks not months. He is on vacation in March so he wants to get it done this far before that.

Talk about Peter's ideas for 3.0 -- Ken Weiner

Peter Kharchenko (who works for Unicon but could not come to the meeting) wants to have a 'portal context' with: a layout generator; more than one rendering pipeline e.g one through SAX filters, one with no transformations; and other services. In other words, have the uPortal be made up of more cleanly delineated snap-together parts, so that the parts are easier to modify/replace

So there needs to be a pretty concise definition of which pieces a particular portal implementation should fire up when initializing. Therefore we should use some standardized way of specifying this constellation of pieces. Our portal.properties file is a rather primitive way of doing this, specifying class names.

Apache's Avalon Merlin -- Peter and Michael looked at this.
Bill Thompson has been looking at the Spring framework.

Some work has already been done on Merlin with uPortal so this is not going to be brought to a halt. Probably soon there will be some parallel implementation with Spring etc.

The Spring Experience -- Bill Thompson (Associate Director, new technology), Dmitriy Kopylenko (Sr. Software Engineer) [Rutgers]

The PowerPoint slides for this presentation.

They picked Spring because of architecture, robustness, and scalability, with respect to focusing on business applications.

Intro to Spring:

J2EE should be easier to use.

Please go look at the JDBC abstraction layer in Spring.

AOP = Aspect Oriented Programming. [a development methodology which considers the manner in which an application does things. AO thinking can best be characterized by sentences beginning with "Whenever". For more information go to the Communications of the ACM archive and look at the October 2001 issue. -- Mike Oltz]

Dmitriy spoke about how they took Pluto and reimplemented the portalImpl stuff with Spring. He showed a UML diagram of making specific Spring-style classes to implement the portalImpl interfaces. He showed a snippet from an XML file to define the beans for the Spring initializer. Then a sequence diagram of how the http requests are handled using Spring.

Is Spring a good fit for uP3?

• Spring is easy to use
• Supports all three IoC types
• No dependencies on framework
• They have found productivity gains

Ken: how many jar files are required for Spring?
Dmitriy: It comes two ways. One enormous jar with everything, or they have several smaller jars so you can pick and choose a subset if there are some major features you don't need.

Recommended book in print: Expert One-on-One J2EE Design and Development by Rod Johnson [Wrox, presumably now Wiley] 0764543857 $59.99

Forthcoming books on Spring:

Spring Live by Matt Raible. Sourcebeat (secure PDF subscription service). May 2004, $29.95

J2EE Development without EJB, Expert One-on-One by Rod Johnson and Juergen Hoeller. Wrox/Wiley, May 2004,
0764558315  $39.99

Professional Spring by Thomas Risberg, Rod Johnson, Juergen Hoeller, Alef Arendsen, plus contributions from several core
Spring developers. [Publication data not available at time of writing this report]

Apache Merlin -- Michael Ivanov

Michael showed a UML diagram of some work he is doing (re Pluto and uPortal 3) and used it as a starting point for his discussion.

Each Merlin "block" can define a context and services to be used in it. Some "lifecycle handlers" are provided and you can write your own such. He spoke about the initialization configuration file, for example how the 'components' refer to particular Java classes, and one component can declare that it requires other components. Another config file outlines the dependencies between components.

Michael then showed the source of several of his classes regarding how the Merlin code works.

[the conversation then turned to discussing the issue in general]

Dan: What were the specific advantages sought in deciding to use a component framework?

Michael: it makes the development better. We don't have to worry about the lifecycle of any objects, because the component framework handles this for us.

Bill: One example the "portal context". The Aggregator, Layout Generator. Using these containers you make it easier to interchange different implementations, and you put the dependencies and so on into the configuration files rather than having to hard code it in the Java. You get a better object model. Better separation of concerns.

The other potential benefit is in the configuration itself. There are a number of properties files and they are growing. Choosing a framework should at least rationalize all that down. Improve configuring.

Alex: It seems the biggest benefit is consistency. It look to me as if the capabilities provided by the frameworks are not anything we couldn't write ourselves.

Bill: At Rutgers we make better architected applications because we're using Spring.

Steve: Are there tools to help you do all this? Editing XML [by hand] is a nightmare. They're not meant to be read by humans. How hard would this be to configure?

Bill: There will be an Eclipse plug-in for Spring.

[Jim Farmer put the issue of frameworks into the context of uPortal and Sakai:]

Jim: One of the fundamental questions is -- should you use a framework?

Re Sakai -- they could choose either Jetspeed or uPortal. It was recommended that they use uPortal because of functionality. But Jetspeed had chosen Apache Avalon Merlin. The conclusion was, since Sakai chose uPortal, and the other project they could have chosen was using a component framework, a framework would be desirable.

Now then, which one? Who makes the decision?

Several mistakes were made. (1) Was this Merlin decision advice, recommendation, or decision? Jim feels he did not make this clear. (2) When should this question have been brought to this group? This question should have been asked of the uportal dev list in November rather than December or January; it's too late to change the initial work.

The question is dollars, risk and time. Cannot delay uPortal past August or preferably July. UMich considers 2.3 to be a fallback in case 3.0 is not ready. Their initial audience is only 1 or 2 thousand.

Alternatives:
1) Continue with Merlin, already 2 months work.
2) Recommend to change over to Spring.
3) What should be discussed in July about frameworks and whether and which?

The reason for 3.0 is that Peter and some others feel that 2.3 won't scale, and it needs tuning and more things put in. The 'more things' were decided in December 2003 at some discussions at the winter JA-Sig conference. And have been mentioned in a slide earlier today (for 2005 implementation). Basically we are relying on Peter Kharchenko's decisions regarding this issue.

Susan: Maybe Peter feels that managing so many components without a framework is too complex.

TUESDAY

Best Practices -- Jim Farmer

In higher ed, there are many choices to make in major software development. How do we apply standards to these projects?

1) Is it appropriate for JA-Sig to have a role in that?
2) How would best practices be defined and advocated? When and under what circumstances?

From the UK there was a project to connect library systems. They chose to add name-value pairs to the end of an existing specification for doing this, because Java cannot handle XML-based extensions to standards (problems with marshalling data for persistence or net transfer). But there were disadvantages to this. Would JA-Sig appropriately have something to say about situations like this?

Mark Boyd: I think Sun [Microsystems, Inc.--a JA-SIG contributor], for example would be more likely to listen to JA-SIG than to a single school.

Chris: We will be ahead of IMS (IMS Global Learning Consortium) in our project. [see Chris's session for the context of this remark.]

Jan: We chose to use the IMS spec. There was a political advantage in marketing, but the way the data is converted for being sent makes it harder to parse it at the other end. Most of the XML standards are focused on standardizing the data format, rather than focused on communicating the data from place to place. There are some portions of the IMS Enterprise that are ambiguous or do not specify features we found to be needed. So we (SCT) extended the standard for what we need.

Jim: The Department of Homeland Security will probably require you to change the way you represent names within the next 12 months. DHS tested 200,000 names against their database, and got 14,000 false positives, because the way the names are represented by the various sources of information is not exact enough.

How do we handle things like what Jan did?
1) Record the changes you made and make them public.
2) During public comment period on the standard, express these concerns to IMS Global.
3) If too overworked, do nothing. :-)

John Bush: the academic people create the standards but then the developers in the trenches find out they're to some degree unworkable.

Chris: There are so many voices speaking to IMS, and the only ones who have a voice are the ones who pay [membership in IMS]. In order for JA-SIG to have a voice, either you'd have to speak through an existing member, or you could have a very public meeting independent of IMS to review and critique the specifications from the point of view of developers, then publish the recommendations.

John Bush: Did you have input into JSR-168?

[no, because JA-SIG has not been in a position for the heavy involvement needed to be in a standards committee.]

Susan: JSR-168 feels like a step backward for uPortal.

Jim: uPortal has been given power because of its deployment.

Howard: We set out to write 'best technology' but JSR-168 sought the lowest common denominator to be able to be inclusive.

[There ensued a debate over whether we were supposed to 'support' 168, as Ken is planning to do in uPortal 2.3, or 'embrace' it (in that it will be the central kind of 'channel' in uPortal 3.0)]

Jan: You have to focus on the particular technology in question. What is the advantage of 168 compared to ours? [Jim made an analogy with error logging] With logging for example, logging is not our core business, so it's not a big deal to throw out personal code and use a standard that has reached some maturity even if we lose flexibility. With using JSR-168 we will be able to write portlets and use other vendors' which is a tremendous advantage.

[someone] What is the disadvantage of JSR-168?

Howard: With XML we can have uniformity and better security. With JSR-168's stream of bytes, the framework cannot look at it and impose uniformity or add security to it. For example we can't handle mixed content (http/https) any more because we can't find the items that need to be proxied.

Jan: We would have to impose additional requirements on other vendors' JSR-168 portlets.

Chris: Then you lose some of the point of having a portlet standard.

Jim: The market does tend to choose less flexible and less well defined standards because they are simpler and more familiar.

Mark: was the point of JSR-168 an attempt by people with an installed base to keep the market in their camp rather than losing people who want to move forward technologically?

Ken: By adding the character stream interface [for rendering channels] we have already lost this flexibility. It feels like everyone is clamoring for JSR-168.

Mark: But we added char stream to support legacy apps.

Ken: ... and for those who didn't care about the flexibility and wanted better performance.

Jim: (1) The market chooses familiar things rather than current technology, and (2) make sure not to forget to think forward, not merely support such a standard.

Howard: The way we think about the portal is, the world is XML and we will parse and render it. The way the others think about portals is, the world is a set of web pages and a portal aggregates them and shows them together.

Jim: A corollary -- we should establish best practices for parties implementing portlets. It looks to me like we're split on whether we should influence things by quietly deploying what we think is best, or by directly participating in standards development. If you're a developer you ought to know or be able to find out how others have done something similarly. If you want to extend IMS Enterprise for example, go find out how others have extended it. A communications point, be sure to communicate to others what you are doing.

Jim: I don't think we could have had an impact on JSR-168 because it was formulated based on what people were doing rather than on what people should be doing.

Dan: Isn't there an inherent ambivalence in our community? (1) A bunch of universities get together to do something the way they want, and it will be cheap. (2) And this is an advantage to corporations and a way to get business for them. For Columbia for example, a portal is not their core business, it's a "filing cabinet." I don't see how you can ever see a harmonic resolution of both perspectives. That's not bad, we just need to manage the dissonance.

Jim: There are people here who want to do something genuinely good, in an environment where good enough will suffice.

Often standards representatives are from the marketing part of their corporations rather than the technical part, even though they may have a technical sounding title.

Dissonance: (1) Quality vs. good enough. (2) Marketing vs. development perspectives.

Again, the best way to deal with the dissonance is communicating.

Jim: Give me recommendations on what to do, and I will present it to the board.

Enterprise Architecture and Components -- Jim Farmer

SAP has made some decisions that have changed the market
1) Use web services to integrate
2) The "packaged composite applications" approach as explained in the following two books:

Packaged Composite Applications: An O'Reilly Field Guide to Enterprise Software by Dan Woods. O'Reilly & Associates; 1st edition (June 2003) ISBN: 0596005520

Enterprise Services Architecture by Dan Woods. O'Reilly & Associates; 1st Edition (September 5, 2003) ISBN: 0596005512

[Jim Farmer comments: "Purchasing these two books is not recommended. The themes and examples would make better short papers than a book. The overall description of the SAP approach is useful. Since SAP does have dominance in the ERP sector, they do have power to define integration].

They developed standards for access to human resources, etc.

If you develop an accounting app for example, would you use the same interfaces as SAP or develop your own?

What interest would there be in JA-SIG in discussing and studying how to do integration?

Steve: Spoke briefly about WebMethods at Cornell.

WebMethods advantages. [by Jim]
1) They developed a tool for automating the design of interfaces to ERP systems.
2) They reverse engineered all the processes in PeopleSoft
3) They put the web services on top of it.

SAP took a similar approach to integration.

[break]

OSPI ePortfolio -- Chris Coppola (r•smart)

Project started late in 2002, officially in Feb 2003. It's a continuation of an electronic portfolio project at U Minnesota for five or six years.

An electronic portfolio -- a web-based application, individual-centric, in which the person can store a bunch of stuff which is samples of their work, or works in progress, for disciplines such as art and archaeology, which are based more on visual or digital media than on text or math. If it were an enterprise application, it would have much more potential because the information can be used for assessment of progress. And do statistical analysis and so on.

The first step was to remove proprietary ties that UMinn had used, such as use of PeopleSoft. In July 2003 the first version came out.

There hasa been a lot of interest in the product. About 850 parties have downloaded the software, 1000 to 1100 people have registered for the online demo.

After July, how would it go foward, how would it get funded? They have a different organizational structure. There is a council (board, re funding and interface to other organizations, procedural questions) and committers (keepers of the s/w archives, make the technical decisions).

The original application started out as an archive to facilitate reflective practices. No assessment or reporting capability. Yet the community that is interested in portfolios finds assessment vital, so that will be the focus of version 2 development. But keep the individual the "center of the universe", unlike course scheduling etc.

Sakai and ePortfolio v2 were funded around the same time and the list of players overlaps somewhat. So what Sakai is doing will probably influence ePortfolio development.

Indiana U and r•smart co. are working together on v2.

OKI OSIDs will be used for v2. r•smart will use the Tool Portability Profile from Sakai. ePortfolio will not use the digital repository part of OSIDs because they feel it is not robust enough. Instead they will use Fedora which is another Mellon funded project.

Jim: OSPI project is important to JA-SIG because 1) For many it will be a standalone application. This is an architectural problem for how it could be available also as channels.
2) Authorization -- this will require document-level authorization. At UMinn, a student could grant another person access to part of their files, even if the person has no affiliation to the institution. [SCT's Documentum has similar capability] [Chris: This will be enhanced in future releases of ePortfolio.]
3) The people who did uPortal had knowledge of what a portal should do. But there is a diversity of opinion as to what a portfolio is and does. The student puts the item in; the teacher comments and shows it to the class; perhaps the student wants to show their work to a potential employer. If you apply electronically, you are two or three weeks ahead, because a third-party company is usually contracted to get the resumes online and it delays the application; the paper form is never read as such.

Chris: Complexity issues in data standards are going to be very important. Portfolios must be portable across multiple institutions and still be individual-centric. For example taking classes at a community college and using the same files at the student's job or internship. That's a future issue for OSPI. This will deploy as standalone, but some institutions will deploy in a portal; some places will have a hybrid deployment. [so he wants us to expose OSIDs outside of the portal.]

Jim: I think it's going to be a 'guerilla' application; that is, it will start quietly in one department's server then gradually grow out to the university. Another example of use of portfolios is nursing training where there are specific projects you must accomplish so it has relevance to certification processes.

[Note by Mike Oltz.. I originally thought Jim said 'gorilla' but he provided this definition:
A "guerrilla application" is one that is installed, say, in a department and becomes an enterprise application because it is adopted
one department or user after another. It would never have been approved as an enterprise system initially. For example, many uPortal implementations are "guerrilla applications." A "gorilla application" would be one that everyone has to have, which is what portals are becoming.]

Susan: are there commercial offerings?

Chris: There are maybe 50 to 100 portfolio applications in all, maybe 14 or so commercially available [Jim: all with sharply different functionality]

Ken: Will you use WSRP?
John: Well we'll be using Web Services at some level.
Ken: 168?
John: Yes, because that's what Sakai is using. We did not feel free to make a different choice.
Ken: they might be able to write WSRP services faster than they could do JSR- 168.
Chris: V2 is going to be a complete rewrite.
Jim: There is a reason for WSRP, because if you do a 'guerilla' application in one department, and it gradually grows, having WSRP puts you in a better position to become mission-critical.

http://www.theospi.org/

Bill Thompson: We also got a request to have something standalone and also as a portal. Has anyone else faced this situation?

Jim: Peter Kharchenko did something like that but it was a long time ago.

Chris: Is there a forum to talk about such [integration] issues between meetings?

Jim: One problem is to notice and identify the issues. How would anybody have known that there is a resumé standard heavily used since last April? There is no mailing list regarding integrating applications.

Bill; Is anyone involved in Open EAI? [U of Illinois] Jan: SCT is.
http://www.openeai.org
Jan: There was a mailing list for developers They wanted to make it open source but it hasn't happened yet.
John: They said they didn't have the resources to support an open source community.

Should the JA-SIG take a role in fostering communication among all these efforts? Or will Sakai do it because it is bigger?

Error Handling

(two proposals for improving error handling in uPortal framework)

Howard Gilbert

1) What is broken in the portal that needs to be fixed?
2) How do you make classes and so on to fix it?

When a top-level method throws an exception, PortalSessionManager would throw ServletException to the container and the end user would see a stacktrace.

What Howard did was to catch it in the SessionManager and trace it on the log. Show a more human message to the user.

Another example, at many places in the uPortal code, getConnection may throw a null, and the next line of code tries to USE it without checking.

When database problems are caught--
in the logon you get a SecurityException
in the groups manager you get a GroupsException
but they're BOTH a database problem.

getProperty() if it can't find the file, no value, value bad format, it throws a RuntimeException.

We didn't have a global strategy for error management. Misunderstanding of java Exception handling followed stylistic models rather than solid software engineering design.

trap all errors at component boundaries.
don't allow any errors to percolate up to the container.
Ditto, don't let anything out of a channel to the framework.
Avoid wardrobe malfunctions -- don't hang out in front of a large audience.
Buck Stops here
Classification --

• who -- end user, configuration, network services, programmer
• what: retry, kill session, kill portal
• when -- try again, try later, try tomorrow
• where -- local, caller, portal, end user
• how: log at throw, log at catch, don't log.

ErrorID objects: Instead of many error classes, have one class and put an errorID into it. You call a particular method from all catch clauses and it does much clerical work, such as trimming down the stacktrace to only show the part in the portal code. rewrite getProperty to provide default values, add a throw clause to throw the PortatlException not a RuntimeException.

[there ensued a debate regarding Howard's contention that a hierarchy of exception classes is bad.]

Jan Nielsen

From Jan's point of view, SCT have taken the traditional Java approach, using a hierarchy of classes, and multiple catch clauses.

Agrees no end user should ever see a stacktrace.

Somebody has to write logic to determine whether the exception was, for example, a database exception or an authentication exception. The way they do this is to have a database abstraction layer. The abstraction layer returns a particular kind of object to indicate the error.

When I design an API with a checked exception, I require the caller to either handle it or explicitly declare that they are not handling it. With an unchecked exception, I don't expect the caller to be able to do anything about it.

[the meeting broke for lunch without resolving the debate. Ken asked Howard and Jan to have lunch together and debate out the issues.]

Ken said that Howard and Jan made progress toward resolving their differences and they will take the issue offline and hope to get a solution plan in a couple of weeks.

Suggestions and ideas from Dan Ellentuck

1) Use Optimized Collections classes

Addresses bugs related to "Concurrency" classes, regarding concurrent access to maps. Dan found concurrency classes written by Doug Lea, that are coming out in JDK 1.5. Presently the package is called EDU.oswego.cs.dl.util.concurrent. Dan thinks they are high quality and easy to use. He specifically looked at the concurrent maps classes.

Compared to collections.synchronizedMap this:

scales much better
High level of concurrency on long running operations
Prevents ConcurrentModificationExceptions under load
Simplifies coding

Disadvantages
Another external dependency
Funky package name [as yet]

Background reading:
Text of JSR-166 spec
An article on the IBM developerworks site by Brian Goetz about the concurrent collections classes implementation.
An overview of the util.concurrent package by Doug Lea.

Trent: Found a bug in his own portal. The javax.naming package says that it is not synchronized. Do the new classes address this?

[answer was -- makes it possible to put an adapter in front that will fix the problem.]

Jan: We use those concurrency classes in our own rendering package. I'm for this.

Bill: We've looked at Doug's code and it has a good reputation. This will relieve uPortal of maintaining its own concurrency classes.

Dan: Not entirely. there are parts of our concurrency that support for example, caching in a multi-server environment, which we will still need.. But this change will simplify the code and make it more robust.

The proposal was accepted.

Dan: "After lunch, everybody agrees with everything."

2) The sequence generator.

The current ones (there are two) use a database. There are collisions and the algorithm can slow the portal to a crawl, especially when creating a new user. We don't use too many keys now but it may increase greatly in the future. One time we considered using a UUID for a key but discarded that.

Dan suggests: We retrieve sequence numbers in blocks, and add a block size to the back end table. Modify the code to hit the database only that so often.

John: What about multi-server usage?

Dan: Well we get the high part of the sequence number from the db and do the low order counting.

Jan: What about gaps?
Dan: Well, there will be gaps.
Bill: Well, there are a lot of numbers.

The proposal was accepted.

3) Groups System Update Bottleneck

Distinguish between problems of the underlying group service, and problems of the clients [e.g. manager channels] that use it.

Traversals of the group system can be:

-- outside-in (member.getContainingGroups())
-- inside-out (group.getMembers())

Our optimizations will address one or the other. Each kind of bottleneck is specific to one or the other.

The chief problem occurs when a group is updated. Most particularly when the user gets their memberships.

Group system rules are enforced when we add a member to a group:
member type must be the same as group type
no circular references*
a member cannot belong to a group more than once (members form a set)*
name of member group must be unique within the containing group*

*tested by calling either group.contains(member) or member.contains(group)

A group discovers if it contains a member by retrieving all of its members and examining them
Pros: Simple, fast after first retrieval
Cons: Painfully slow on large (1000s of members) groups

fix:

Add 2 methods to store and service api, delegate contains() to service and/or store, and defer initializing a group with its members until they are explicitly requested.

Issues:

1. If members of a group form a set, why not allow adding a member to a group that already contains it? Membership is unchanged. don't consider it an error, just don't add them a second time.

2. implementation of containsGroupNamed() is potentially very expensive. Must member group names be unique? How about groups from different services? The problem is if a group contains foreign groups you have to fetch all the members of the foreign group then ask each member for its name. not always possible to ask the foreign group 'give me the member which has such a name' and just look whether it returned an item.

Jan: we implement our group store like in issue 1. So it's not necessary to for the caller to check for duplication.

We agreed to change group system rules as follows:

• Adding the same member to a group more than once has no affect, but is not an error.
• Adding a group with a duplicate name to a group is now permitted.  In other words, group "Buddies" may now contain multiple groups named "My Buddies" so long as each group has a unique key.

4) When does it make sense to lock a group for update?

when you add a group to another group you need to lock to prevent circular references. Do you need to lock when adding leaves? Just prevent race conditions in the client (the caller).

Trent: If a client tries to reference a group while another thread is trying to delete the same group, the client might get the group anyway. But the second time the client tried to get that group it would not work.

New Group Store based on IPerson Attributes -- Ken Weiner and Dan Ellentuck

"Person Attributes Group Service" (PAGS)

Ken -- people want to create a group called 'Students'. They can detect via personattributes how to set up the person. Should be possible later to query whether the user is a student to drive authorization. But the way the group service is written now, you'd have to get the whole list of students from LDAP and then see if the person is in it. Instead, why not just look directly into the person attributes from the group system. The present group system has no handles to IPerson objects. The group service was written to be independent of the person system, so it has user keys not IPerson stuff.

Dan: -- how do we feel about sharing IPerson attributes, and the issues of privacy?

Howard: [who wrote the personDirectory service] the current PersonDirectory stuff reads from one or more LDAP servers and one or more databases. You could always write code to look in there and decide what kind of user you have. "The principle was always there." Another info source that could be used is a SAML assertion.

Steve: This would make possible an implementation of authorization "policies".

Jan: Luminus generally does not use the group structure because we had an existing infrastructure based on LDAP that scaled with our needs. We faced the same issue. It's a tradeoff between acquiring a sufficient number of attributes off the user object initially to be able to make later decisions, versus getting as thin a user as possible initially and then force later actual queries to go to the server. We chose the latter approach to make logins as fast as possible.

Ken: With respect to uPortal, we have already read all the attributes, so should we create a group service that can access them?

Trent: Is this basically a performance issue?

Susan: It's also a flexibility issue.

Trent: What we did was to add LDAP attributes, such as 'courses'.

Dan: But this is to hit the Person Attributes, not specifically LDAP.

Jan: We have two kinds of groups, "static groups" where you specifically list the members in it. and "filters" that you hit against the directory server and build dynamically.

[the general proposal was accepted]

Ken asked how it should be done. How should the group service be able to get IPerson? Should the PersonDirectory be extended to have a static method to get the IPerson for a given person? Some would prefer that you could only get the attributes map, because there could be the Security Context in there, and that might give arbitrary code 'access to the world' for that person.

Steve: Why not put some meaningful methods in IPerson?

Ken: Extend IPerson that has no security context in it, give THAT version to the group service.

Dan: Another requirement is to have multi-valued attributes.
Howard: Several people have tinkered this by putting in a Vector, but this breaks various places that assumes the result is a String.
Ken: Add a new method that would return a multi-valued thing. Add another parm in the XML that tells whether the particular store supports multi-value. If you call the old method on a multi-valued thing, you will get back what? Decision: the first attribute.

Dan: This kind of design requires that you know in advance what group you want, and under what circumstances a person is a member of the group, and then you can associate this with a permission.

Steve: We put in caching so that if the LDAP server goes down and the user comes back for a later session, they still get the IPerson.

[break]

Agenda review for work items -- Ken

2.3 will be out, "my best guess is March" but it will be plus or minus a month. It depends on whether he gets done with his part before or after vacation. According to Jim, 3.0 needs to be out in August. Ken does not think that there is anything other than Sakai dictating this. Ken doesn't think it can be finished by then. He is not ready to commit to anything as yet.

Jan: What functional changes do you see between 3.0 beyond performance improvements?
Ken: The point is to be able to swap major rendering components; it's not relevant to performance. This will enable later changes such as handling detached channels. In general it will be the same portal with some functional enhancements.

The Pluto section is all that has to be in 2.3. Everything else is at best 3.0 and may be later.

[the following are Mike Oltz's notes about this session; the official list of items and assignments is that prepared by Ken]

Aggregated Layouts:
UI for managing pushed fragments --
Improve UI for content subscriber - pulled fragments -- Shawn Lonas

Integrated modes theme: improve usability of the UI -- e.g. graying out unusable controls
More than two skins -- Shawn Lonas
Portlet CSS implementation -- New set of CSS classes for JSR-168 and WSRP then convert existing portal UI to use those skins -- Shawn Lonas

Migration Instructions and Tools -- Susan Bramhall, Steve Barrett, Al Wold, one of the UNICON people

Internationalization:
Mapping resource bundles into XSLT -- Alex Vigdor, Shoji (XSLT changes)
Session-based locale selector
UI for choosing sorted list of locales

WSRP:
Testing and bug fixing - Steve Barrett
Registration interface - Ken Weiner, Kevin Gary
Authentication? (nebulous at this point)
Consumer URL rewriting
-- consumer gives producer info, producer rewrites -- Ken did this.
-- producer sends templates to consumer, consumer rewrites -- Nick Bolton

Pluto: [this is for 2.3]
Static information provider service - Michael Ivanov, Mark Boyd
Dynamic info provider service - Michael Ivanov and ken Weiner
Factory service - Michael Ivanov
Deployment process - going from a portlet class to a packaged distribution -- Mark Boyd

IoC Frameworks [this is for 3.0]
Configure Pluto container with service components (with Merlin) -- Michael Ivanov
[Ken would like someone try to mirror the effort in Spring]
Bill -- let's discuss on the mail list at what point the parallel effort begins and how.
Ken -- Peter should comment on that question. In the next month or so there should be enough converted to be able to begin the comparison.
Howard -- do we stop using properties files a la J2EE and go to the component framework way, or do we provide a parallel way to configure via the framework and leave the original ways also.

Doug Lea's concurrency packages -- Dan Ellentuck
Improve sequence generator -- Dan Ellentuck
Groups improvements -- Dan Ellentuck
Person Attributes Group Service -- Dan Ellentuck and Ken Weiner and Al Wold
multi-valued person attributes -- Al Wold and Howard Gilbert [there is a Bugzilla item requesting this feature]
FERPA concerns [issue was not raised before this point, it was brought up in a personal conversation during a break] -- Dan Ellentuck and Steve Valero
Group service paging mechanism [like in a database] so you would get a specific number of group members at a time, then ask for more -- Dan Ellentuck, Mark Boyd, Dave Wallace, Al Wold

Error Handling:
Howard and Jan to continue discussion (current plan is to make an undeclared PortalException that will just percolate up until it hits a catch somewhere) i18n of error messages -- Jan Nielsen

Howard: Tomcat 5 is Servlet spec 2.4 which has new methods that the older HTTPServletRequest doesn't have. Compilation problems. -- Howard Gilbert

Xerces distributes two different jars. One with DOM2, and an experimental DOM3. We have PortalDocumentImpl in the utils, and guess what it breaks against DOM3.
Michael Ivanov: It's possible to use reflection to tell which one you have.

Support DOM3 in our Document impl -- Nick Bolton.

Dan: I'm very concerned about the OKI issue. We need to hear more concretely what we expect the relationship between the OKI and the uPortal API's will be, so we can react to that.

Ken: We called Sakai and talked on the phone but that didn't work well, we got confused. So we asked them for a paper document with a diagram of the interfaces they expect to see. They haven't provided that yet; we need to remind them we're waiting for that.

Howard: Suppose that 2.3 is really close, and 3.0 is however far off. Will Sakai require something else to be put in in between? Will we need a 2.4 release?

Jim: Which do you do, propose a solution because it meets our understanding of the requirements, or do we wait to get a definition and respond to it? I think they haven't said anything because they really don't know yet. [In yesterday's first session, Jim had a slide in which] I put most of the things on the above list, into 2005. There's no guarantee they'll get done soon. Indiana U. has some things that they need to migrate their portal onto uPortal, and those things are not on the 2.3 or 3.0 list.