WSLCB - Integrator Work Session
(October 29, 2020) - Summary

Akerna - Money Printer Go Brrrrr
[ Event Details ]

IT staff confirmed an unchanged status quo with the State’s seed-to-sale traceability vendor MJ Freeway, an Akerna company, and discussed additional ways to workaround the system.

Here are some observations from the Thursday October 29th Washington State Liquor and Cannabis Board (WSLCB) Integrator Work Session.

My top 3 takeaways:

  • WSLCB information technology (IT) staff confirmed an unchanged status quo with the State’s seed-to-sale traceability vendor MJ Freeway, an Akerna company, while providing updates to software integrators on system workarounds.
    • WSLCB Integrator Work Sessions convene agency technical staff and third-party software providers which interface with the State's cannabis supply chain traceability system, Leaf Data Systems. MJ Freeway representatives are sometimes present on the calls, but were not for this particular meeting. The last work session occurred on August 27th.
    • Agency staff present included:
      • Mary Mueller, Chief Information Officer (CIO)
      • Kendra Hodgson, Cannabis Examiner Manager
      • Joseph Kelly, System Administrator
      • Rocky Atwood, System Administrator
      • Terri Noble, Administrative Assistant to the CIO
      • Gerard Scheeler, Systems Administrator
      • Robert Villemaire, Cannabis Examiner
    • Software integrators and observers:
    • Kelly welcomed members (audio - 2m) before “jump[ing] into some updates on the status of efforts” (audio - 5m).
      • Kelly reported on “on-going efforts to improve the Leaf system” which were “limited to workflows and changes only to the interface file.” Coding changes weren’t “planned, and currently none are being authorized at this time,” he stated.
      • Staff continued working on the “licensing assumptions processing workaround” which was in a “testing phase, and has been fighting resourcing issues.” He elaborated, “most of the [WSLCB] positions designated to supporting Leaf are vacant” and the agency was engaged in a “resourcing conversation.”
      • Kelly said they were preparing to “decommission...the test environment where we will be syncing the user data from test to training.” This would also update “the training environment to the latest production version.”
      • Efforts were underway to “define the dissolution of business process” which would require licensees to “do some administrative work in the Leaf system prior to their business losing the licensed status.”
      • Kelly mentioned new workflows for “wet flower transfer” as well as “pesticide and heavy metals compliance” were being created by WSLCB’s Examiner Team as workarounds to achieve “system compliance.”
      • Kelly confirmed the Traceability 2.0 effort was “on hold.”
      • He also made clear that use of the magic WAATESTE “workaround for lab results was not in compliance with the directive of LCB.” Businesses that “utilize that workaround to get lab results into the system” were admonished to “stop doing that.”
      • Kelly reiterated that integrators were expected to “submit support tickets to MJ Freeway when there are issues” to enable agency staff to attempt to coordinate with the vendor on the problem.
    • Mueller provided updates on the State’s strained relationship with the vendor and operational constraints which put future planning on hold.
      • She began by acknowledging the twelfth contract amendment between the agency and MJ Freeway signed in June, and indicated its boilerplate language could “be revisited” during the next cycle in December as WSLCB continued working on “several escalated issues…particularly around data fixes.” She had been led to believe that staffing issues at MJ Freeway impacted “their ability” to address issues. “Response times” had been discussed with the vendor and WSLCB was “actively pursuing resolutions” (audio - 2m).
      • Regarding the Traceability 2.0 effort, Mueller spoke to challenges agency staff had advancing the project due to budget shortfalls and a state hiring freeze. She reaffirmed the agency still supported recommendations proposed in the Traceability 2.0 work group, and were considering how to pursue them (audio - 1m).
  • Software integrators and attendees had several questions about MJ Freeway as well as a potential new workaround to track pesticide and heavy metals test results.
    • Regarding continual problems working with MJ Freeway, Busby wanted to know why “it feels like we keep hearing that every single month,” and asked what “mechanisms” were available to get the company “to do their job.” Mueller expressed “full faith” that WSLCB leadership and the Washington State Office of the Attorney General (OAG) planned to leverage “the provisions that are provided to us in the contract.” Busby pressed on who at the agency was able to talk about contract negotiations, and Mueller responded that Deputy Director Megan Duffy and Director Rick Garza led that effort. Busby asked if it was reasonable for integrators to expect the “the same nothing that we’ve had” for the preceding year, and Mueller sympathized with his frustration (audio - 3m).
      • The day before the work session, MJ Freeway’s parent company Akerna announced “a public offering of 5,000,000 shares of its common stock at a public offering price of $2.40 per share” and subsequently confirmed “gross proceeds of approximately $12,000,000.”
    • Towery returned to the topic of the WAATESTE workaround and asked for authoritative documentation “about what the expected behavior of lab inheritance is.” He believed this would reduce use of WAATESTE as well as “actually point out the defects in Leaf Data [Systems].” Kelly responded that lab result inheritance was an on-going discussion with MJ Freeway representatives, and proposed creating a new work group to include Towery, the agency, and the vendor. Towery, Casey, Busby, and others on the call voiced interest in participation. Busby lamented that “we’re restricted by policy from sharing correct information with each other.” After further feedback, Kelly admitted “we are fairly certain” WAATESTE continued to be available through an unidentified integrator’s software, “so I’m asking all the integrators to clean house, check your systems, make sure it's not you.” Multiple participants seemed to indicate they knew exactly which third-party software vendor was at fault, but none made a public statement to that effect. Mueller promised that another notice against using the workaround would be issued by WSLCB (audio - 16m).
    • Farsi asked about workarounds dealing with product returns (audio - 4m).
    • Foster followed up on a previous suggestion to collect data on MJ Freeway’s support efforts such as “turnaround time” to gauge their “performance” in relation to the State’s generous contract. He expected the $50,000 monthly fee paid by WSLCB largely funded MJ Freeway’s customer support efforts since development was non-existent and operational costs would be minimal. Mueller said it had been asked, but she was unaware of any response or data which resulted. She indicated Kelly was privy to a “weekly report on how long things have been” but the contract did not require an “actual effort report” or hours worked by the company supporting the traceability system (audio - 3m).
    • Foster then asked for “technical details” about the development of new workarounds to support pesticide and heavy metals test results to understand “what those are going to look like” (audio - 5m).
      • Mueller stated the workarounds had yet to “go through the agency process” before being released.
      • Hodgson elaborated that the workarounds were being prepared in anticipation of changes from the Quality Control (QC) Testing and Product Requirements rulemaking project underway at WSLCB. Staff had “put in a ticket on how to use existing workflow to accommodate the change in the policy” and she acknowledged it remained a possibility changes would “not be able to be accommodated.” Hodgson was firm that the testing standards and policies would exist even in the event “the system cannot record that information.” In the event a workaround couldn’t be achieved, she said the agency would give a “records retention requirement” for licensees to document pesticide and heavy metals testing and “demonstrate you meet those standards.”
      • Foster wanted to know if “expecting progress on this front” was reasonable prior to the rulemaking project’s public hearing on November 18th. Hodgson answered that while no new information was forecast before that hearing, the listed effective dates were “staggered” throughout 2021, with the final effective date in February 2022. “The policy is the driving force on this,” Hodgson said, “the technology is not going to be the thing that stops this.”
    • Farsi asked a final question about migration of a licensee from a different integration provider (audio - 4m).
  • Before concluding the meeting, staff reiterated the diminished capacity of WSLCB’s IT department and the escalation path for identified issues.
    • Mueller provided more context about the 160 software applications used at the agency which the IT department was responsible for, remarking that “each of those have multiple business processes associated with them.” Her team was navigating decreased funding and the fact that all of WSLCB’s “dedicated tester” positions were vacant “and we’re in a hiring freeze.” Kelly was still dedicating “a lot of time” to traceability, Mueller stated, “but the limited resources that we have are dedicated to the projects and efforts that the agency prioritizes.” Asked about the potential for layoffs by Foster, she acknowledged that the IT division would be “reducing five positions” at the end of the fiscal year, and “every single division” at the agency would experience “significant” staffing cutbacks (audio - 5m).
    • Kelly received a chat comment about a filed ticket which he used as a process example, saying it had been escalated to Duffy who would take the ticket up with MJ Freeway. Mueller chimed in to say if the agency determined they couldn’t address a ticket they passed it on to vendor staff regularly and “if there’s an issue to escalate, [Duffy] escalates it immediately.” Mueller noted that staff attempted to convene an "engineering meeting" with MJ Freeway the prior week but it was scuttled due to “internal confusion with the vendor.” Nonetheless, WSLCB staff engaged in a conversation with a senior engineer at MJ Freeday to convey a litany of issues and “why it was important that these things get root cause and resolution as soon as possible.” Mueller believed that agency representatives had “given everything that we can to them and now it's just about delivering back to us.” Her hope was that a “final engineering meeting” to resolve already identified issues would occur the following week. She remained committed to do “everything that we think that we can without putting the industry further at risk” (audio - 3m).
    • Mueller closed out the meeting asking integrators to keep filing tickets with the vendor and sharing them with WSLCB at wslcb@sp.lcb.wa.gov if they needed to be escalated. Kelly added that getting “the runaround” from MJ Freeway was “a cause to reach out to us, too” (audio - 1m).

Information Set