by Joche Ojeda | Oct 15, 2024 | A.I, Semantic Kernel, XAF, XPO
A few weeks ago, I forked the Semantic Kernel repository to experiment with it. One of my first experiments was to create a memory provider for XPO. The task was not too difficult; basically, I needed to implement the IMemoryStore interface, add some XPO boilerplate code, and just like that, we extended the Semantic Kernel memory store to support 10+ databases. You can check out the code for the XpoMemoryStore here.
My initial goal in creating the XpoMemoryStore was simply to see if XPO would be a good fit for handling embeddings. Spoiler alert: it was! To understand the basic functionality of the plugin, you can take a look at the integration test here.
As you can see, usage is straightforward. You start by connecting to the database that handles embedding collections, and all you need is a valid XPO connection string:
using XpoMemoryStore db = await XpoMemoryStore.ConnectAsync("XPO connection string");
In my original design, everything worked fine, but I faced some challenges when trying to use my new XpoMemoryStore in XAF. Here’s what I encountered:
- The implementation of XpoMemoryStore uses its own data layer, which can lead to issues. This needs to be rewritten to use the same data layer as XAF.
- The XpoEntry implementation cannot be extended. In some use cases, you might want to use a different object to store the embeddings, perhaps one that has an association with another object.
To address these problems, I introduced the IXpoEntryManager interface. The goal of this interface is to handle object creation and queries.
public interface IXpoEntryManager
{
T CreateObject();
public event EventHandler ObjectCreatedEvent;
void Commit();
IQueryable GetQuery(bool inTransaction = true);
void Delete(object instance);
void Dispose();
}
Now, object creation is handled through the CreateObject<T> method, allowing the underlying implementation to be changed to use a UnitOfWork or ObjectSpace. There’s also the ObjectCreatedEvent event, which lets you access the newly created object in case you need to associate it with another object. Lastly, the GetQuery<T> method enables redirecting the search for records to a different type.
I’ll keep updating the code as needed. If you’d like to discuss AI, XAF, or .NET, feel free to schedule a meeting: Schedule a Meeting with us.
Until next time, XAF out!
Related Article
https://www.jocheojeda.com/2024/09/04/using-the-imemorystore-interface-and-devexpress-xpo-orm-to-implement-a-custom-memory-store-for-semantic-kernel/
by Joche Ojeda | Jan 22, 2024 | Sqlite
SQLite and Its Journal Modes: Understanding the Differences and Advantages
SQLite, an acclaimed lightweight database engine, is widely used in various applications due to its simplicity, reliability, and open-source nature. One of the critical aspects of SQLite that ensures data integrity and supports various use-cases is its “journal mode.” This mode is a part of SQLite’s transaction mechanism, which is vital for maintaining database consistency. In this article, we’ll explore the different journal modes available in SQLite and their respective advantages.
Understanding Journal Modes in SQLite
Journal modes in SQLite are methods used to handle transactions and rollbacks. They dictate how the database engine logs changes and how it recovers in case of failures or rollbacks. There are several journal modes available in SQLite, each with unique characteristics suited for different scenarios.
1. Delete Mode
Description:
The default mode in SQLite, Delete mode, creates a rollback journal file alongside the database file. This file records a copy of the original unchanged data before any modifications.
Advantages:
- Simplicity: Easy to understand and use, making it ideal for basic applications.
- Reliability: It ensures data integrity by preserving original data until the transaction is committed.
2. Truncate Mode
Description:
Truncate mode operates similarly to Delete mode, but instead of deleting the journal file at the end of a transaction, it truncates it to zero length.
Advantages:
- Faster Commit: Reduces the time to commit transactions, as truncating is generally quicker than deleting.
- Reduced Disk Space Usage: By truncating the file, it avoids leaving large, unused files on the disk.
3. Persist Mode
Description:
In Persist mode, the journal file is not deleted or truncated but is left on the disk with its header marked as inactive.
Advantages:
- Reduced File Operations: This mode minimizes file system operations, which can be beneficial in environments where these operations are expensive.
- Quick Restart: It allows for faster restarts of transactions in busy systems.
4. Memory Mode
Description:
Memory mode stores the rollback journal in volatile memory (RAM) instead of the disk.
Advantages:
- High Performance: It offers the fastest possible transaction times since memory operations are quicker than disk operations.
- Ideal for Temporary Databases: Best suited for databases that don’t require data persistence, like temporary caches.
5. Write-Ahead Logging (WAL) Mode
Description:
WAL mode is a significant departure from the traditional rollback journal. It writes changes to a separate WAL file without changing the original database file until a checkpoint occurs.
Advantages:
- Concurrency: It allows read operations to proceed concurrently with write operations, enhancing performance in multi-user environments.
- Consistency and Durability: Ensures data integrity and durability without locking the entire database.
6. Off Mode
Description:
This mode disables the rollback journal entirely. Transactions are not atomic in this mode.
Advantages:
- Maximum Speed: It can be faster since there’s no overhead of maintaining a journal.
- Use Case Specific: Useful for scenarios where speed is critical and data integrity is not a concern, like intermediate calculations or disposable data.
Conclusion
Choosing the right journal mode in SQLite depends on the specific requirements of the application. While Delete and Truncate modes are suitable for most general purposes, Persist and Memory modes serve niche use-cases. WAL mode stands out for applications requiring high concurrency and performance. Understanding these modes helps developers and database administrators optimize SQLite databases for their particular needs, balancing between data integrity, speed, and resource utilization.
In summary, SQLite’s flexibility in journal modes is a testament to its adaptability, making it a preferred choice for a wide range of applications, from embedded systems to web applications.
